@databricks/zerobus-ingest-sdk 1.1.0 → 1.3.0

package/README.md

@@ -28,7 +28,7 @@ The Databricks Zerobus Ingest SDK for TypeScript provides a high-performance cli
 - **High-throughput ingestion**: Optimized for high-volume data ingestion with native Rust implementation
 - **Automatic recovery**: Built-in retry and recovery mechanisms for transient failures
 - **Flexible configuration**: Customizable stream behavior and timeouts
-- **Multiple serialization formats**: Support for JSON, Protocol Buffers, and Arrow Flight (Beta) with optional LZ4 / ZSTD compression
+- **Multiple serialization formats**: Support for JSON, Protocol Buffers, and Arrow Flight ingestion with optional LZ4 / ZSTD compression
 - **Type widening**: Accept high-level types (plain objects, protobuf messages) or low-level types (strings, buffers) - automatically handles serialization
 - **Batch ingestion**: Ingest multiple records with a single acknowledgment for higher throughput
 - **OAuth 2.0 authentication**: Secure authentication with client credentials
@@ -42,21 +42,14 @@ The Databricks Zerobus Ingest SDK for TypeScript provides a high-performance cli
 - **Node.js**: >= 16
 - **Databricks workspace** with Zerobus access enabled
 
-### Build Requirements
+### Source Build Requirements
 
 - **Rust toolchain**: 1.70 or higher - [Install Rust](https://rustup.rs/)
 - **Cargo**: Included with Rust
+- Platform C/C++ build tools
 
-### Dependencies
-
-These will be installed automatically:
-
-```json
-{
-  "@napi-rs/cli": "^2.18.4",
-  "napi-build": "^0.3.3"
-}
-```
+You only need these source-build tools when npm cannot use a pre-built native
+package for your platform, or when developing the SDK from this repository.
 
 ## Quick Start User Guide
 
@@ -66,108 +59,30 @@ Before using the SDK, you need a Databricks workspace URL, a Delta table, and a
 
 ### Installation
 
-#### Prerequisites
-
-Before installing the SDK, ensure you have the required tools:
-
-**1. Node.js >= 16**
-
-Check if Node.js is installed:
 ```bash
-node --version
+npm install @databricks/zerobus-ingest-sdk
 ```
 
-If not installed, download from [nodejs.org](https://nodejs.org/).
-
-**2. Rust Toolchain (1.70+)**
+On supported platforms, npm installs the TypeScript package and the matching
+pre-built native binary package automatically.
 
-The SDK requires Rust to compile the native addon. Install using `rustup` (the official Rust installer):
+#### Local Development From Source
 
-**On Linux and macOS:**
-```bash
-curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
-```
-
-Follow the prompts (typically just press Enter to accept defaults).
-
-**On Windows:**
-
-Download and run the installer from [rustup.rs](https://rustup.rs/), or use:
-```powershell
-# Using winget
-winget install Rustlang.Rustup
-
-# Or download from https://rustup.rs/
-```
-
-**Verify Installation:**
-```bash
-rustc --version
-cargo --version
-```
+Clone and build from source only when modifying this SDK or when your platform
+does not have a pre-built native binary:
 
-You should see version 1.70 or higher. If the commands aren't found, restart your terminal or add Rust to your PATH:
 ```bash
-# Linux/macOS
-source $HOME/.cargo/env
-
-# Windows (PowerShell)
-# Restart your terminal
+git clone https://github.com/databricks/zerobus-sdk.git
+cd zerobus-sdk/typescript
+npm install
+npm run build
 ```
 
-**Additional Platform Requirements:**
-
-- **Linux**: Build essentials
-  ```bash
-  # Ubuntu/Debian
-  sudo apt-get install build-essential
-
-  # CentOS/RHEL
-  sudo yum groupinstall "Development Tools"
-  ```
-
-- **macOS**: Xcode Command Line Tools
-  ```bash
-  xcode-select --install
-  ```
-
-- **Windows**: Visual Studio Build Tools
-  - Install [Visual Studio Build Tools](https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2022)
-  - During installation, select "Desktop development with C++"
-
-#### Installation Steps
-
-1. Clone the repository:
-   ```bash
-   git clone https://github.com/databricks/zerobus-sdk.git
-   cd zerobus-sdk/ts
-   ```
-
-2. Install dependencies:
-   ```bash
-   npm install
-   ```
-
-3. Build the native addon:
-   ```bash
-   npm run build
-   ```
-
-   This will compile the Rust code into a native Node.js addon (`.node` file) for your platform.
-
-4. Verify the build:
-   ```bash
-   # You should see a .node file
-   ls -la *.node
-   ```
-
-5. The SDK is now ready to use! You can:
-   - Use it directly in this directory for examples
-   - Link it globally: `npm link`
-   - Or copy it into your project's `node_modules`
-
 **Troubleshooting:**
 
+- **Unsupported platform or source build requested**: Install Rust 1.70+,
+  Cargo, and your platform C/C++ build tools, clone this repository, and run
+  `npm install` followed by `npm run build` from `zerobus-sdk/typescript`
 - **"rustc: command not found"**: Restart your terminal after installing Rust
 - **Build fails on Windows**: Ensure Visual Studio Build Tools are installed with C++ support
 - **Build fails on Linux**: Install build-essential or equivalent package
@@ -175,13 +90,18 @@ source $HOME/.cargo/env
 
 ### Choose Your Serialization Format
 
-The SDK supports two serialization formats. **Protocol Buffers is the default** and recommended for production use:
+The SDK supports three ingestion formats. Protocol Buffers is the default record format and recommended for production row-oriented workloads:
 
 - **Protocol Buffers (Default)** - Strongly-typed schemas, efficient binary encoding, better performance. This is the default format.
 - **JSON** - Simple, no schema compilation needed. Good for getting started quickly or when schema flexibility is needed.
+- Arrow Flight - High-performance columnar ingestion for applications that already produce Arrow data. See the [Arrow example](examples/arrow/README.md).
 
 > **Note:** If you don't specify `recordType`, the SDK will use Protocol Buffers by default. To use JSON, explicitly set `recordType: RecordType.Json`.
 
+### Acknowledgments and throughput
+
+Ingestion is asynchronous. `ingestRecordOffset()` (and `ingestRecordsOffset()`) resolves as soon as the record is queued; the SDK sends it and tracks its acknowledgment in the background. To confirm records are durably committed, call `flush()` — it resolves once everything queued so far is acknowledged. The idiomatic flow is **ingest in a loop, then `flush()`** (once for a bounded batch, or periodically for a long-running stream). Each ingest also returns the record's offset, and `waitForOffset(offset)` resolves when that offset is acknowledged — handy when a specific record must be confirmed before continuing (acks are ordered, so waiting on the last offset confirms the whole run). Just avoid calling `waitForOffset()` after every record in a tight loop, since that limits throughput to one record per round-trip. The examples below follow this pattern.
+
 ### Option 1: Using JSON (Quick Start)
 
 JSON mode is the simplest way to get started. You don't need to define or compile protobuf schemas, but you must explicitly specify `RecordType.Json`.
@@ -189,12 +109,13 @@ JSON mode is the simplest way to get started. You don't need to define or compil
 ```typescript
 import { ZerobusSdk, RecordType } from '@databricks/zerobus-ingest-sdk';
 
+async function main(): Promise<void> {
 // Configuration
 // For AWS:
 const zerobusEndpoint = 'https://<workspace-id>.zerobus.<region>.cloud.databricks.com';
 const workspaceUrl = 'https://<workspace-name>.cloud.databricks.com';
 // For Azure:
-// const zerobusEndpoint = '<workspace-id>.zerobus.<region>.azuredatabricks.net';
+// const zerobusEndpoint = 'https://<workspace-id>.zerobus.<region>.azuredatabricks.net';
 // const workspaceUrl = 'https://<workspace-name>.azuredatabricks.net';
 
 const tableName = 'main.default.air_quality';
@@ -223,8 +144,6 @@ const stream = await sdk.createStream(
 );
 
 try {
-    let lastOffset: bigint;
-
     // Send all records
     for (let i = 0; i < 100; i++) {
         const record = {
@@ -233,16 +152,23 @@ try {
             humidity: 50 + (i % 40)
         };
 
-        // ingestRecordOffset returns immediately after queuing
-        lastOffset = await stream.ingestRecordOffset(record);
+        // Queue the record; do not wait for its acknowledgement here
+        await stream.ingestRecordOffset(record);
     }
 
     // Wait for all records to be acknowledged
-    await stream.waitForOffset(lastOffset);
+    await stream.flush();
     console.log('Successfully ingested 100 records!');
 } finally {
     await stream.close();
 }
+
+}
+
+main().catch((error) => {
+  console.error('Fatal error:', error);
+  process.exitCode = 1;
+});
 ```
 
 ### Option 2: Using Protocol Buffers (Default, Recommended)
@@ -254,7 +180,15 @@ Protocol Buffers is the default serialization format and provides efficient bina
 Before starting, ensure you have:
 
 1. **Protocol Buffer Compiler (`protoc`)** - Required for generating descriptor files
-2. **protobufjs** and **protobufjs-cli** - Already included in package.json devDependencies
+2. **protobufjs** - Required at runtime by your generated Protocol Buffer code
+3. **protobufjs-cli** - Required during development to generate JavaScript and type declarations
+
+Install the JavaScript runtime and code-generation tools in your application:
+
+```bash
+npm install protobufjs
+npm install --save-dev protobufjs-cli
+```
 
 #### Step 1: Install Protocol Buffer Compiler
 
@@ -295,7 +229,9 @@ protoc --version
 
 #### Step 2: Define Your Protocol Buffer Schema
 
-The SDK includes an example schema at `schemas/air_quality.proto`:
+Create `schemas/air_quality.proto` in your application with the following
+example schema. Also create an `examples/generated` directory for the generated
+JavaScript and type declarations:
 
 ```protobuf
 syntax = "proto2";
@@ -315,13 +251,8 @@ message AirQuality {
 Generate TypeScript code from your proto schema:
 
 ```bash
-npm run build:proto
-```
-
-This runs:
-```bash
-pbjs -t static-module -w commonjs -o examples/generated/air_quality.js schemas/air_quality.proto
-pbts -o examples/generated/air_quality.d.ts examples/generated/air_quality.js
+npx pbjs -t static-module -w commonjs -o examples/generated/air_quality.js schemas/air_quality.proto
+npx pbts -o examples/generated/air_quality.d.ts examples/generated/air_quality.js
 ```
 
 **Output:**
@@ -351,8 +282,9 @@ That's it! The SDK will automatically extract the message descriptor from this f
 ```typescript
 import { ZerobusSdk, RecordType } from '@databricks/zerobus-ingest-sdk';
 import * as airQuality from './examples/generated/air_quality';
-import { loadDescriptorProto } from '@databricks/zerobus-ingest-sdk/utils/descriptor';
+import { loadDescriptorProto } from '@databricks/zerobus-ingest-sdk/utils/descriptor.js';
 
+async function main(): Promise<void> {
 // Configuration
 const zerobusEndpoint = 'https://<workspace-id>.zerobus.<region>.cloud.databricks.com';
 const workspaceUrl = 'https://<workspace-name>.cloud.databricks.com';
@@ -388,26 +320,32 @@ const stream = await sdk.createStream(tableProperties, clientId, clientSecret, o
 
 try {
     const AirQuality = airQuality.examples.AirQuality;
-    let lastOffset: bigint;
 
     // Send all records
     for (let i = 0; i < 100; i++) {
         const record = AirQuality.create({
-            device_name: `sensor-${i}`,
+            deviceName: `sensor-${i}`,
             temp: 20 + i,
             humidity: 50 + i
         });
 
-        // ingestRecordOffset returns immediately after queuing
-        lastOffset = await stream.ingestRecordOffset(record);
+        // Queue the record; do not wait for its acknowledgement here
+        await stream.ingestRecordOffset(record);
     }
 
     // Wait for all records to be acknowledged
-    await stream.waitForOffset(lastOffset);
+    await stream.flush();
     console.log('Successfully ingested 100 records!');
 } finally {
     await stream.close();
 }
+
+}
+
+main().catch((error) => {
+  console.error('Fatal error:', error);
+  process.exitCode = 1;
+});
 ```
 
 #### Type Mapping: Delta ↔ Protocol Buffers
@@ -466,24 +404,16 @@ message NestedData {
    EOF
    ```
 
-2. **Add build script to package.json:**
-   ```json
-   {
-     "scripts": {
-       "build:proto:myschema": "pbjs -t static-module -w commonjs -o examples/generated/my_schema.js schemas/my_schema.proto && pbts -o examples/generated/my_schema.d.ts examples/generated/my_schema.js"
-     }
-   }
-   ```
-
-3. **Generate code and descriptor:**
+2. **Generate code and descriptor:**
    ```bash
-   npm run build:proto:myschema
+   npx pbjs -t static-module -w commonjs -o examples/generated/my_schema.js schemas/my_schema.proto
+   npx pbts -o examples/generated/my_schema.d.ts examples/generated/my_schema.js
    protoc --descriptor_set_out=schemas/my_schema_descriptor.pb --include_imports schemas/my_schema.proto
    ```
 
-4. **Load descriptor in your code:**
+3. **Load descriptor in your code:**
    ```typescript
-   import { loadDescriptorProto } from '@databricks/zerobus-ingest-sdk/utils/descriptor';
+   import { loadDescriptorProto } from '@databricks/zerobus-ingest-sdk/utils/descriptor.js';
    const descriptorBase64 = loadDescriptorProto({
        descriptorPath: 'schemas/my_schema_descriptor.pb',
        protoFileName: 'my_schema.proto',
@@ -497,7 +427,7 @@ message NestedData {
 - Install `protoc` (see Step 1 above)
 
 **"Cannot find module './generated/air_quality'"**
-- Run `npm run build:proto` to generate TypeScript code
+- Run the `npx pbjs` and `npx pbts` commands from Step 3
 
 **"Descriptor file not found"**
 - Generate the descriptor file using the commands in Step 4
@@ -509,27 +439,27 @@ message NestedData {
 - Make sure you're using `loadDescriptorProto()` from the utils
 
 **Build fails on proto generation**
-- Ensure protobufjs is installed: `npm install --save-dev protobufjs protobufjs-cli`
+- Ensure the runtime and CLI are installed: `npm install protobufjs` and
+  `npm install --save-dev protobufjs-cli`
 
 #### Quick Reference
 
-Complete setup from scratch:
+After creating `schemas/air_quality.proto` and the `examples/generated`
+directory as described above:
 ```bash
-# Install dependencies and build SDK
-npm install
-npm run build
+# Install the SDK and protobuf codegen tools
+npm install @databricks/zerobus-ingest-sdk protobufjs
+npm install --save-dev protobufjs-cli
 
-# Setup Protocol Buffers
-npm run build:proto
+# Generate protobuf code and descriptor
+npx pbjs -t static-module -w commonjs -o examples/generated/air_quality.js schemas/air_quality.proto
+npx pbts -o examples/generated/air_quality.d.ts examples/generated/air_quality.js
 protoc --descriptor_set_out=schemas/air_quality_descriptor.pb --include_imports schemas/air_quality.proto
-
-# Run example
-npm run example:proto:single
 ```
 
 #### Why Two Steps (TypeScript + Descriptor)?
 
-1. **TypeScript Code Generation** (`npm run build:proto`):
+1. **TypeScript Code Generation** (`npx pbjs` and `npx pbts`):
    - Creates JavaScript/TypeScript code for your application
    - Provides type-safe message creation and encoding
    - Used in your application code
@@ -543,10 +473,14 @@ Both are necessary for Protocol Buffers ingestion!
 
 ## Usage Examples
 
-See the `examples/` directory for complete, runnable examples. See [examples/README.md](examples/README.md) for detailed instructions.
+The source repository contains complete, runnable examples in `examples/`.
+Clone and build the repository using the [local development](#local-development-from-source)
+instructions, then see [examples/README.md](examples/README.md) for details.
 
 ### Running Examples
 
+Run these commands from the cloned repository's `typescript` directory:
+
 ```bash
 # Set environment variables
 export ZEROBUS_SERVER_ENDPOINT="https://<workspace-id>.zerobus.<region>.cloud.databricks.com"
@@ -576,7 +510,7 @@ For higher throughput, use batch ingestion to send multiple records with a singl
 
 ```typescript
 const records = Array.from({ length: 1000 }, (_, i) =>
-  AirQuality.create({ device_name: `sensor-${i}`, temp: 20 + i, humidity: 50 + i })
+  AirQuality.create({ deviceName: `sensor-${i}`, temp: 20 + i, humidity: 50 + i })
 );
 
 // Protobuf Type 1: Message objects (high-level) - SDK auto-serializes
@@ -659,7 +593,7 @@ const stream = await sdk.createStream(
   '', // client_secret (ignored when headers_provider is provided)
   options,
   {
-    getHeadersCallback: async () => [
+    getHeadersCallback: () => [
       ["authorization", `Bearer ${myToken}`],
       ["x-databricks-zerobus-table-name", tableName]
     ]
@@ -680,7 +614,7 @@ const stream = await sdk.createStream(
 | Option | Default | Description |
 |--------|---------|-------------|
 | `recordType` | `RecordType.Proto` | Serialization format: `RecordType.Json` or `RecordType.Proto` |
-| `maxInflightRequests` | 10,000 | Maximum number of unacknowledged requests |
+| `maxInflightRequests` | 1,000,000 | Maximum number of unacknowledged requests |
 | `recovery` | true | Enable automatic stream recovery |
 | `recoveryTimeoutMs` | 15,000 | Timeout for recovery operations (ms) |
 | `recoveryBackoffMs` | 2,000 | Delay between recovery attempts (ms) |
@@ -714,13 +648,15 @@ const stream = await sdk.createStream(
 ## Descriptor Utilities
 
 The SDK provides a helper function to extract Protocol Buffer descriptors from FileDescriptorSets.
+Use the `.js` subpath shown below for compatibility with CommonJS and native
+Node.js ESM imports.
 
 ### loadDescriptorProto()
 
 Extracts a specific message descriptor from a FileDescriptorSet:
 
 ```typescript
-import { loadDescriptorProto } from '@databricks/zerobus-ingest-sdk/utils/descriptor';
+import { loadDescriptorProto } from '@databricks/zerobus-ingest-sdk/utils/descriptor.js';
 
 const descriptorBase64 = loadDescriptorProto({
     descriptorPath: 'schemas/my_schema_descriptor.pb',
@@ -753,38 +689,37 @@ const descriptorBase64 = loadDescriptorProto({
 
 ## Error Handling
 
-The SDK includes automatic recovery for transient failures (enabled by default with `recovery: true`). For permanent failures, use `recreateStream()` to automatically recover all unacknowledged batches. Always use try/finally blocks to ensure streams are properly closed:
+The SDK includes automatic recovery for transient failures (enabled by default with `recovery: true`). `getUnackedBatches()` and `recreateStream()` succeed only after a terminal native-stream failure, which already closes the stream. An enqueue failure leaves the wrapper active, so those calls reject; rethrow the original error. Do not call `stream.close()` before `recreateStream()`, because close releases the native handle.
 
 ```typescript
+let replacement;
 try {
     const offset = await stream.ingestRecordOffset(record);
-    await stream.waitForOffset(offset);
+    await stream.flush();
     console.log(`Success: offset ${offset}`);
 } catch (error) {
     console.error('Ingestion failed:', error);
-
-    // When stream fails, close it first
-    await stream.close();
-    console.log('Stream closed after error');
-
-    // Optional: Inspect what needs recovery (must be called on closed stream)
-    const unackedBatches = await stream.getUnackedBatches();
-    console.log(`Batches to recover: ${unackedBatches.length}`);
-
-    // Recommended recovery approach: Use recreateStream()
-    // This method:
-    // 1. Gets all unacknowledged batches from the failed stream
-    // 2. Creates a new stream with the same configuration
-    // 3. Re-ingests all unacknowledged batches automatically
-    // 4. Returns the new stream ready for continued use
-    const newStream = await sdk.recreateStream(stream);
-    console.log(`Stream recreated with ${unackedBatches.length} batches re-ingested`);
-
-    // Continue using newStream for further ingestion
     try {
-        // Continue ingesting...
+        const unackedBatches = await stream.getUnackedBatches();
+        console.log(`Batches to recover: ${unackedBatches.length}`);
+        replacement = await sdk.recreateStream(stream);
+        await replacement.flush();
+    } catch (recoveryError) {
+        console.error('Stream was not terminal or recovery failed:', recoveryError);
+        throw new AggregateError(
+            [error, recoveryError],
+            'ingestion and recovery both failed',
+        );
     } finally {
-        await newStream.close();
+        if (replacement) {
+            await replacement.close();
+        }
+    }
+} finally {
+    try {
+        await stream.close();
+    } catch (closeError) {
+        console.error('Failed stream released:', closeError);
     }
 }
 ```
@@ -804,12 +739,14 @@ Main entry point for the SDK.
 **Constructor:**
 
 ```typescript
-new ZerobusSdk(zerobusEndpoint: string, unityCatalogUrl: string)
+new ZerobusSdk(zerobusEndpoint: string, unityCatalogUrl: string, options?: ZerobusSdkOptions)
 ```
 
 **Parameters:**
 - `zerobusEndpoint` (string) - The Zerobus gRPC endpoint (e.g., `https://<workspace-id>.zerobus.<region>.cloud.databricks.com` for AWS, or `https://<workspace-id>.zerobus.<region>.azuredatabricks.net` for Azure)
 - `unityCatalogUrl` (string) - The Unity Catalog endpoint (your workspace URL)
+- `options` (ZerobusSdkOptions, optional) - Additional SDK configuration:
+  - `applicationName` (string, optional) - Application identifier appended to the HTTP `user-agent` header, conventionally `"<product>/<version>"` (e.g. `"my-app/1.0"`). The header becomes `zerobus-sdk-ts/<version> <applicationName>`, enabling server-side attribution.
 
 **Methods:**
 
@@ -845,23 +782,29 @@ This method is the **recommended approach** for recovering from stream failures.
 4. Returns the new stream ready for continued ingestion
 
 **Parameters:**
-- `stream` - The failed or closed stream to recreate
+- `stream` - The terminally failed stream to recreate. Do not call `stream.close()`
+  first because the TypeScript wrapper releases its native handle on close.
 
 **Returns:** Promise resolving to a new `ZerobusStream` with all unacknowledged batches re-ingested
 
 **Example:**
 ```typescript
 try {
-  await stream.ingestRecords(batch);
+  await stream.ingestRecordsOffset(batch);
+  await stream.flush();
 } catch (error) {
-  await stream.close();
-  // Automatically recreate stream and recover all unacked batches
+  // recreateStream() rejects unless the native stream already failed closed.
   const newStream = await sdk.recreateStream(stream);
-  // Continue ingesting with newStream
+  try {
+    await newStream.flush();
+  } finally {
+    await newStream.close();
+  }
 }
 ```
 
-**Note:** This method preserves batch structure and re-ingests batches atomically. For debugging, you can inspect what was recovered using `getUnackedBatches()` after closing the stream.
+**Note:** This method preserves batch structure and re-ingests batches atomically. For
+debugging, inspect `getUnackedBatches()` after a terminal failure and before closing the wrapper.
 
 ---
 
@@ -875,13 +818,16 @@ Represents an active ingestion stream.
 async ingestRecordOffset(payload: Buffer | string | object): Promise<bigint>
 ```
 
-**(Recommended)** Ingests a single record. The Promise resolves immediately after the record is queued (before server acknowledgment). Use `waitForOffset()` to wait for acknowledgment when needed.
+**(Recommended)** Ingests a single record. The Promise resolves immediately after the record is queued (before server acknowledgment); the round-trip happens in the background. The idiomatic flow is to ingest in a loop and then `flush()` once to confirm everything queued so far. The returned offset, together with `waitForOffset()`, lets you confirm a specific record when needed — prefer that for bulk over waiting after each record, since per-record waiting limits throughput to one round-trip per record.
 
 ```typescript
-// High-throughput pattern: send many, wait once
-const offset1 = await stream.ingestRecordOffset(record1);  // Resolves immediately
-const offset2 = await stream.ingestRecordOffset(record2);  // Resolves immediately
-await stream.waitForOffset(offset2);  // Waits for server to acknowledge all records up to offset2
+// Idiomatic flow: ingest in a loop, then flush once
+let lastOffset: bigint | null = null;
+for (const record of records) {
+  lastOffset = await stream.ingestRecordOffset(record);  // Resolves immediately
+}
+await stream.flush();  // Resolves once everything queued so far is acknowledged
+// (Or, to confirm a specific record: if (lastOffset !== null) await stream.waitForOffset(lastOffset))
 ```
 
 ---
@@ -890,7 +836,7 @@ await stream.waitForOffset(offset2);  // Waits for server to acknowledge all rec
 async ingestRecordsOffset(payloads: Array<Buffer | string | object>): Promise<bigint | null>
 ```
 
-**(Recommended)** Ingests multiple records as a batch. The Promise resolves immediately after the batch is queued (before server acknowledgment). Returns `null` for empty batches.
+**(Recommended)** Ingests multiple records as a batch. The Promise resolves immediately after the batch is queued (before server acknowledgment); the round-trip happens in the background. Returns `null` for empty batches. As with `ingestRecordOffset()`, the idiomatic flow is to ingest in a loop and `flush()` once to confirm; reach for `waitForOffset()` when a specific batch must be confirmed before continuing.
 
 ---
 
@@ -898,7 +844,7 @@ async ingestRecordsOffset(payloads: Array<Buffer | string | object>): Promise<bi
 async waitForOffset(offsetId: bigint): Promise<void>
 ```
 
-Waits for the server to acknowledge all records up to and including the specified offset ID.
+Waits for the server to acknowledge all records up to and including the specified offset ID. Acks are ordered, so waiting on the **last** offset confirms every prior record too. Use this when a specific record must be confirmed before continuing; for confirming a bulk run, `flush()` is usually simpler. Avoid calling it after every record in a tight loop, since that limits throughput to one record per round-trip.
 
 ---
 
@@ -998,7 +944,7 @@ await stream.ingestRecords(buffers);
 async flush(): Promise<void>
 ```
 
-Flushes all pending records and waits for acknowledgments.
+Flushes all pending records and waits for acknowledgments. This is the recommended way to confirm a batch of `ingestRecordOffset()` / `ingestRecordsOffset()` calls: ingest in a loop without waiting, then `flush()` once at the end instead of calling `waitForOffset()` after every record.
 
 ```typescript
 async close(): Promise<void>
@@ -1012,7 +958,8 @@ async getUnackedRecords(): Promise<Buffer[]>
 
 Returns unacknowledged record payloads as a flat array for inspection purposes.
 
-**Important:** Can only be called on **closed streams**. Call `stream.close()` first, or this will throw an error.
+**Important:** This can only be called after a terminal stream failure. Do not call
+`stream.close()` first: the TypeScript wrapper releases the underlying stream handle on close.
 
 **Returns:** Array of Buffer containing the raw record payloads
 
@@ -1026,7 +973,8 @@ async getUnackedBatches(): Promise<Buffer[][]>
 
 Returns unacknowledged records grouped by their original batches for inspection purposes.
 
-**Important:** Can only be called on **closed streams**. Call `stream.close()` first, or this will throw an error.
+**Important:** This can only be called after a terminal stream failure. Do not call
+`stream.close()` first: the TypeScript wrapper releases the underlying stream handle on close.
 
 **Returns:** Array of arrays, where each inner array represents a batch of records as Buffers
 
@@ -1039,15 +987,11 @@ try {
   await stream.ingestRecords(batch2);
   // ... error occurs
 } catch (error) {
-  await stream.close();
   const unackedBatches = await stream.getUnackedBatches();
   // unackedBatches[0] contains records from batch1 (if not acked)
   // unackedBatches[1] contains records from batch2 (if not acked)
 
-  // Re-ingest with new stream
-  for (const batch of unackedBatches) {
-    await newStream.ingestRecords(batch);
-  }
+  console.log(`Batches available for recovery: ${unackedBatches.length}`);
 }
 ```
 
@@ -1070,10 +1014,10 @@ interface TableProperties {
 
 ```typescript
 // JSON mode
-const tableProperties = { tableName: 'main.default.air_quality' };
+const jsonTableProperties = { tableName: 'main.default.air_quality' };
 
 // Protocol Buffers mode
-const tableProperties = {
+const protoTableProperties = {
     tableName: 'main.default.air_quality',
     descriptorProto: descriptorBase64  // Required for protobuf
 };
@@ -1090,7 +1034,7 @@ Configuration options for stream behavior.
 ```typescript
 interface StreamConfigurationOptions {
     recordType?: RecordType;              // RecordType.Json or RecordType.Proto. Default: RecordType.Proto
-    maxInflightRequests?: number;         // Default: 10,000
+    maxInflightRequests?: number;         // Default: 1,000,000
     recovery?: boolean;                   // Default: true
     recoveryTimeoutMs?: number;           // Default: 15,000
     recoveryBackoffMs?: number;           // Default: 2,000
@@ -1110,11 +1054,12 @@ enum RecordType {
 
 1. **Reuse SDK instances**: Create one `ZerobusSdk` instance per application
 2. **Stream lifecycle**: Always close streams in a `finally` block to ensure all records are flushed
-3. **Batch size**: Adjust `maxInflightRequests` based on your throughput requirements (default: 10,000)
+3. **Batch size**: Adjust `maxInflightRequests` based on your throughput requirements (default: 1,000,000)
 4. **Error handling**: The stream handles errors internally with automatic retry. Only use `recreateStream()` for persistent failures after internal retries are exhausted.
 5. **Use Protocol Buffers for production**: Protocol Buffers (the default) provides better performance and schema validation. Use JSON only when you need schema flexibility or for quick prototyping.
 6. **Store credentials securely**: Use environment variables, never hardcode credentials
 7. **Use batch ingestion**: For high-throughput scenarios, use `ingestRecordsOffset()` instead of individual `ingestRecordOffset()` calls
+8. **Ingest in a loop, then `flush()`**: See [Acknowledgments and throughput](#acknowledgments-and-throughput) above for the full explanation.
 
 ## Platform Support
 
@@ -1161,7 +1106,7 @@ This SDK wraps the high-performance [Rust Zerobus SDK](https://github.com/databr
 **Benefits:**
 - **Native performance** - Rust implementation for high-throughput ingestion
 - **Native async/await support** - Rust futures become JavaScript Promises
-- **Automatic memory management** - No manual cleanup required
+- **Automatic memory management** for native objects. You still must `await stream.close()` to flush and release the stream.
 - **Type safety** - Compile-time checks on both sides
 
 ## Community and Contributing