@loxtep/sdk 0.1.0

package/docs/sdk-ingestion-export-guide.md

@@ -0,0 +1,480 @@
+# SDK Ingestion and Export Architecture Guide
+
+This guide explains how the Loxtep SDK enables programmatic data ingestion
+(writing events) and export (reading events) through the platform's data mesh.
+It covers the architecture, data flows, authentication, and error handling you
+need to build reliable SDK-based pipelines.
+
+## Table of Contents
+
+- [Architecture Overview](#architecture-overview)
+- [Key Concepts](#key-concepts)
+- [Control Plane vs Data Plane](#control-plane-vs-data-plane)
+- [Data Flow: Write Path (Ingestion)](#data-flow-write-path-ingestion)
+- [Data Flow: Read Path (Export)](#data-flow-read-path-export)
+- [Authentication](#authentication)
+- [Error Handling](#error-handling)
+- [Related Documentation](#related-documentation)
+
+---
+
+## Architecture Overview
+
+The SDK connector loop connects five core entities:
+
+| Entity | Role |
+|--------|------|
+| **SDK_Connector** | Organization-level connector record (`connector_type: "sdk"`) that provides configuration for the SDK client. No OAuth or API polling — just config. |
+| **Trigger / Export** | Workflow-scoped nodes that reference an SDK_Connector by `connector_id`. A **Trigger** represents inbound data (writing events in), an **Export** represents outbound data (reading events out). |
+| **Workflow** | A project-scoped graph of triggers, transformations, validations, and exports that defines how data moves through the platform. |
+| **Data Product** | The published output of a workflow — a governed, discoverable dataset backed by one or more event queues. |
+| **Stream_Bus** | The RStreams/Leo event bus (Kinesis, Firehose, DynamoDB) used by the SDK data plane for writing and reading events. |
+
+### How They Relate
+
+```mermaid
+graph TB
+    subgraph "Control Plane (REST API)"
+        UI[Platform Frontend]
+        API[Connectors Microservice<br/>POST/GET /connectors]
+        MCP[Customer MCP Server<br/>loxtep_connectors]
+        DB[(PostgreSQL<br/>connectors table)]
+        S3T[(S3 Template Catalog<br/>sdk/connector.json)]
+    end
+
+    subgraph "Data Plane (Stream Bus)"
+        SDK_W[SDK Client<br/>FlowWriter]
+        SDK_R[SDK Client<br/>QueueReader]
+        KIN[Kinesis / Firehose]
+        Q[(Event Queue<br/>Data Product)]
+    end
+
+    UI -->|POST /connectors| API
+    MCP -->|POST /connectors| API
+    API -->|INSERT| DB
+    API -->|Load template| S3T
+    API -->|Return sdk_config| UI
+    API -->|Return sdk_config| MCP
+
+    SDK_W -->|putEvents via RStreams| KIN
+    KIN -->|Ingest| Q
+    SDK_R -->|offloadEvents via RStreams| Q
+
+    UI -.->|Copy config| SDK_W
+    MCP -.->|Provide config| SDK_W
+```
+
+The **control plane** (left) handles connector creation, workflow management,
+and configuration. The **data plane** (right) handles the actual event
+read/write through the stream bus. The SDK client bridges both: it uses the
+REST API to fetch configuration, then writes and reads events directly on the
+stream bus.
+
+---
+
+## Key Concepts
+
+### SDK Connector
+
+An SDK connector is fundamentally different from OAuth or API-key connectors:
+
+- **No external credentials to test** — `POST /connectors/{id}/test` returns
+  `{ passed: true }` immediately.
+- **No polling schedule or webhook endpoint** — the SDK client pushes and pulls
+  events on demand.
+- **Metadata carries SDK config** — the connector's `metadata.sdk_config`
+  contains everything needed to bootstrap `LoxtepClient`:
+
+```json
+{
+  "api_url": "https://api.loxtep.io",
+  "organization_id": "<uuid>",
+  "project_id": "<uuid>",
+  "instance_id": "<uuid>",
+  "region": "us-east-1"
+}
+```
+
+### Trigger and Export Nodes
+
+Workflow trigger and export nodes store `connector_id` and project-specific
+config like `output_queue_name` and `bot_id`. These nodes live in S3 under
+`workflows/{workflow_id}/triggers/{id}.json` or
+`workflows/{workflow_id}/exports/{id}.json`.
+
+**Trigger node** (inbound — writing events into the platform):
+
+```json
+{
+  "id": "trigger-001",
+  "type": "sdk",
+  "connector_id": "<connector_uuid>",
+  "configuration": {
+    "output_queue_name": "my-ingestion-queue",
+    "bot_id": "my-writer-bot",
+    "instance_id": "<instance_uuid>",
+    "region": "us-east-1"
+  }
+}
+```
+
+**Export node** (outbound — reading events out of the platform):
+
+```json
+{
+  "id": "export-001",
+  "type": "sdk",
+  "connector_id": "<connector_uuid>",
+  "configuration": {
+    "queue_name": "my-output-queue",
+    "bot_id": "my-reader-bot"
+  }
+}
+```
+
+### FlowWriter and QueueReader
+
+These are the SDK's data plane primitives:
+
+- **FlowWriter** — buffers events and writes them in batches to the stream bus
+  via `putPayloadsToQueue()`. Supports configurable batch size, flush interval,
+  and retry with exponential backoff.
+- **QueueReader** — reads events from a queue via `readQueueBatch()` and yields
+  them as an `AsyncIterable<QueueEvent>`.
+
+---
+
+## Control Plane vs Data Plane
+
+The Loxtep platform separates concerns into two planes. Understanding this
+distinction is essential for configuring the SDK correctly.
+
+### Control Plane (REST API)
+
+The control plane handles all management operations through SigV4-signed HTTP
+requests with a JWT token (`x-jwt-token` header).
+
+| Operation | Endpoint | Description |
+|-----------|----------|-------------|
+| Create connector | `POST /connectors` | Create an SDK connector record |
+| Get connector | `GET /connectors/{id}` | Retrieve connector with `sdk_config` |
+| Test connector | `POST /connectors/{id}/test` | Always returns `{ passed: true }` for SDK |
+| List connector types | `GET /connector-types` | Includes `"sdk"` when template is loaded |
+| Deploy workflow | `POST /workflows/{id}/deploy` | Deploy a workflow graph |
+| Query data products | `GET /data-products` | List and search data products |
+| Replay events | `POST /data-products/{id}/replay` | Replay historical events over HTTP |
+
+**Auth model:** JWT (org/user scoped) + AWS SigV4 signing. The JWT carries
+organization and user identity; SigV4 authenticates the request to API Gateway.
+
+### Data Plane (Stream Bus)
+
+The data plane handles live event read/write through the RStreams/Leo event bus.
+This is a separate infrastructure layer from the REST API.
+
+| Operation | SDK Method | Bus Operation |
+|-----------|-----------|---------------|
+| Write events | `flows.get_writer()` → `writer.write()` → `writer.close()` | `putPayloadsToQueue()` via Kinesis/Firehose |
+| Read events | `queues.open_reader()` → `reader.read()` | `readQueueBatch()` via DynamoDB |
+| Stream live | `data_products.stream()` | `offloadEvents()` via bus |
+
+**Auth model:** AWS IAM credentials scoped to the stream infrastructure
+(DynamoDB, Kinesis, S3). The platform JWT does **not** grant bus access
+directly — bus access uses AWS principals. See
+[sdk-bus-rbac-threat-model.md](./sdk-bus-rbac-threat-model.md) for the full
+security model.
+
+### When Each Plane Is Used
+
+```
+┌─────────────────────────────────────────────────────────┐
+│  Your Application                                       │
+│                                                         │
+│  const client = new LoxtepClient({ ... });              │
+│                                                         │
+│  // Control plane (REST API)                            │
+│  const connector = await client.connectors.get(id);     │
+│  const products = await client.data_products.list();    │
+│  await client.data_products.replay(dpId, opts);         │
+│                                                         │
+│  // Data plane (Stream Bus)                             │
+│  const writer = await client.flows.get_writer(flowId, { │
+│    bot_id: 'my-bot',                                    │
+│    output_queue_name: 'my-queue'                        │
+│  });                                                    │
+│  writer.write({ payload: 'event-data' });               │
+│  await writer.close();                                  │
+│                                                         │
+│  const reader = await client.queues.open_reader({       │
+│    bot_id: 'my-bot',                                    │
+│    queue_name: 'my-queue'                               │
+│  });                                                    │
+│  for await (const event of reader.read()) { ... }       │
+└─────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Data Flow: Write Path (Ingestion)
+
+When your application writes events through the SDK, data flows through these
+stages:
+
+```mermaid
+sequenceDiagram
+    participant App as User Application
+    participant SDK as LoxtepClient
+    participant FW as FlowWriter
+    participant Bus as RStreams SDK
+    participant KIN as Kinesis/Firehose
+    participant Q as Event Queue
+
+    App->>SDK: flows.get_writer(flow_id, { bot_id, output_queue_name })
+    SDK->>SDK: Resolve stream bus config
+    SDK->>FW: Create FlowWriter(rsdk, bot_id, queue)
+    App->>FW: writer.write(event)
+    FW->>FW: Buffer event
+    App->>FW: writer.close()
+    FW->>Bus: putPayloadsToQueue(buffered_events)
+    Bus->>KIN: Write batch
+    KIN->>Q: Deliver to queue
+```
+
+### Step by Step
+
+1. **Get a writer** — call `client.flows.get_writer(flowId, options)` with a
+   `bot_id` (identifies your writer process for checkpointing) and
+   `output_queue_name` (the target queue in your data product).
+
+2. **Write events** — call `writer.write(event)` for each event. Events are
+   buffered in memory. The buffer flushes automatically when it reaches
+   `batch_size` (default: 100) or after `flush_interval_ms` (default: 5000ms).
+
+3. **Close the writer** — call `await writer.close()` to flush all remaining
+   buffered events and confirm delivery. Always close the writer when done to
+   avoid data loss.
+
+4. **Delivery** — the FlowWriter calls `putPayloadsToQueue()` on the RStreams
+   SDK, which writes the batch to Kinesis/Firehose. Events are then delivered
+   to the target event queue (data product).
+
+### Write Path Configuration
+
+```typescript
+const writer = await client.flows.get_writer(flowId, {
+  bot_id: 'my-ingestion-bot',       // Required: identifies this writer
+  output_queue_name: 'raw-events',  // Required: target queue name
+  batch_size: 100,                  // Optional: events per batch (default: 100)
+  flush_interval_ms: 5000,          // Optional: auto-flush interval (default: 5000)
+  max_retries: 3,                   // Optional: retry attempts (default: 3)
+});
+```
+
+### Retry Behavior
+
+The FlowWriter retries transient write failures with exponential backoff:
+
+| Attempt | Delay | Total Elapsed |
+|---------|-------|---------------|
+| 1 | 0ms | 0ms |
+| 2 | 1000ms | 1000ms |
+| 3 | 2000ms | 3000ms |
+| Fail | — | Throw `StreamingError` |
+
+Only transient errors (network timeouts, throttling, 5xx responses) trigger
+retries. Non-transient errors (4xx, auth failures) fail immediately.
+
+---
+
+## Data Flow: Read Path (Export)
+
+When your application reads events from a data product queue:
+
+```mermaid
+sequenceDiagram
+    participant App as User Application
+    participant SDK as LoxtepClient
+    participant QR as QueueReader
+    participant Bus as RStreams SDK
+    participant Q as Event Queue
+
+    App->>SDK: queues.open_reader({ bot_id, queue_name })
+    SDK->>SDK: Resolve stream bus config
+    SDK->>QR: Create QueueReader(rsdk, bot_id, queue)
+    App->>QR: reader.read()
+    QR->>Bus: readQueueBatch(bot_id, queue, batch_size)
+    Bus->>Q: Read batch
+    Q-->>Bus: Events
+    Bus-->>QR: QueueEvent[]
+    QR-->>App: AsyncIterable<QueueEvent>
+```
+
+### Step by Step
+
+1. **Open a reader** — call `client.queues.open_reader({ bot_id, queue_name })`
+   with a `bot_id` (identifies your reader for checkpointing) and the
+   `queue_name` to read from.
+
+2. **Read events** — call `reader.read()` which returns an
+   `AsyncIterable<QueueEvent>`. Events are yielded in the order they were
+   written, in batches of `batch_size` (default: 100).
+
+3. **Close the reader** — call `reader.close()` when done to release resources.
+
+### Alternative Read Methods
+
+| Method | Use Case |
+|--------|----------|
+| `queues.open_reader()` | Low-level queue access with bot checkpointing |
+| `data_products.stream()` | Stream live events from a data product (requires `bot_id`) |
+| `data_products.replay()` | Replay historical events over the REST API (no bus config needed) |
+
+`data_products.replay()` is a control-plane operation — it reads historical
+events via HTTP, not the stream bus. This is useful when you don't have bus
+credentials configured or need to replay a specific time range.
+
+---
+
+## Authentication
+
+The SDK uses a layered authentication model. Control plane and data plane have
+separate auth mechanisms.
+
+### Control Plane Auth (JWT)
+
+The SDK resolves JWT tokens in this precedence order:
+
+1. **`LOXTEP_AUTH_TOKEN` environment variable** — highest priority. Set this
+   for CI/CD pipelines, containers, or any environment where you want explicit
+   token control.
+
+2. **`~/.loxtep/credentials.json`** — written by `loxtep login`. Used for
+   local development. The SDK's TokenManager handles automatic refresh when
+   tokens expire.
+
+If neither source provides a token, API calls fail with `AuthenticationError`.
+
+### Setting Up Auth
+
+**Option A: Environment variable (recommended for CI/CD)**
+
+```bash
+export LOXTEP_AUTH_TOKEN="your-jwt-token"
+```
+
+**Option B: Interactive login (recommended for local dev)**
+
+```bash
+npx loxtep login
+# Stores credentials at ~/.loxtep/credentials.json
+# Prints confirmation on success
+```
+
+**Option C: Config file**
+
+After running `loxtep init`, the SDK reads defaults from
+`~/.loxtep/config.json`:
+
+```json
+{
+  "api_url": "https://api.loxtep.io",
+  "organization_id": "<uuid>",
+  "project_id": "<uuid>",
+  "instance_id": "<uuid>",
+  "region": "us-east-1"
+}
+```
+
+These values serve as defaults for `LoxtepClient` constructor options that are
+not explicitly provided.
+
+### Data Plane Auth (AWS IAM)
+
+Stream bus operations use AWS IAM credentials, not the JWT. The SDK resolves
+stream bus configuration through:
+
+1. **`LoxtepClient({ streams: { ... } })` constructor option** — explicit
+   config.
+2. **`LOXTEP_RSTREAMS_CONFIG_FILE` env var** — path to a JSON file with bus
+   resource names.
+3. **`LEO_*` / `AWS_REGION` env vars** — merged by `resolveStreamsConfiguration`.
+4. **`client.observe.stream_config()`** — automatic resolution from the
+   platform API (calls `GET /observe/stream-config`).
+
+See [sdk-rest-vs-bus.md](./sdk-rest-vs-bus.md) for the full configuration
+surface and security model.
+
+### Exporting Config from a Connector
+
+After creating an SDK connector, export its configuration to bootstrap the
+client:
+
+```bash
+# Shell format (default) — copy-paste into terminal
+loxtep config export --from-connector <connector_id>
+
+# JSON format — for programmatic use
+loxtep config export --from-connector <connector_id> --format json
+
+# .env format — for dotenv files
+loxtep config export --from-connector <connector_id> --format env
+```
+
+---
+
+## Error Handling
+
+### Common Failure Modes
+
+| Error Condition | Error Class | Message | Resolution |
+|----------------|-------------|---------|------------|
+| No auth token available | `AuthenticationError` | "No authentication token found. Set LOXTEP_AUTH_TOKEN or run `loxtep login`" | Set the `LOXTEP_AUTH_TOKEN` env var or run `loxtep login` to store credentials. |
+| Token expired | `AuthenticationError` | "Authentication token expired. Run `loxtep login` to refresh" | Run `loxtep login` again. The TokenManager handles auto-refresh for credentials-file tokens, but env-var tokens must be replaced manually. |
+| Stream bus not configured | `StreamingError` | "Stream bus configuration missing. Set LEO_* environment variables or add `streams` to ~/.loxtep/config.json" | Configure stream bus resources via env vars, config file, or `client.observe.stream_config()`. See [sdk-rest-vs-bus.md](./sdk-rest-vs-bus.md). |
+| Queue not found | `NotFoundError` | "Queue '{queue_name}' not found. Verify the queue exists in your data product" | Check the queue name matches a queue in your data product. Verify the workflow is deployed. |
+| Bot unauthorized | `AuthorizationError` | "Bot '{bot_id}' does not have read permission on queue '{queue_name}'" | Ensure the bot has the correct IAM permissions for the stream bus resources. |
+| Write failed after retries | `StreamingError` | "Failed to write events after 3 attempts: {last_error}" | Check network connectivity and stream bus health. Transient errors (throttling, 5xx) are retried automatically; persistent failures indicate an infrastructure issue. |
+| Connector not found (CLI) | `NotFoundError` | "Connector '{connector_id}' not found" | Verify the connector ID is correct and belongs to your organization. |
+| Connector not SDK type (CLI) | `ValidationError` | "Connector '{connector_id}' is type '{type}', not 'sdk'" | Use `--from-data-product` for non-SDK connectors, or create an SDK connector first. |
+
+### Error Hierarchy
+
+```
+LoxtepError (base)
+├── AuthenticationError    — missing or expired tokens
+├── AuthorizationError     — insufficient permissions
+├── NotFoundError          — resource does not exist
+├── ValidationError        — invalid input or configuration
+└── StreamingError         — stream bus write/read failures
+```
+
+### Debugging Tips
+
+1. **Check auth first** — most failures trace back to missing or expired
+   tokens. Run `loxtep login` and verify `~/.loxtep/credentials.json` exists.
+
+2. **Verify stream config** — if write/read operations fail with
+   `StreamingError`, check that `LEO_*` env vars or `streams` config point to
+   the correct instance. Use `client.observe.stream_config()` to auto-resolve.
+
+3. **Match instance and bus** — the stream bus resources (DynamoDB tables,
+   Kinesis streams, S3 buckets) are 1:1 with a Loxtep instance. Wrong instance
+   config means wrong table names, silent misrouting, or hard failures.
+
+4. **Check bot permissions** — `AuthorizationError` on read/write means the
+   `bot_id` lacks IAM permissions for the target queue's underlying AWS
+   resources.
+
+---
+
+## Related Documentation
+
+| Document | Description |
+|----------|-------------|
+| [SDK Control vs Data Plane](./sdk-control-vs-data-plane.md) | Detailed breakdown of control plane and data plane responsibilities |
+| [SDK REST vs Bus](./sdk-rest-vs-bus.md) | Configuration surfaces, security model, and credential handling |
+| [SDK Bus RBAC Threat Model](./sdk-bus-rbac-threat-model.md) | Security analysis of bus access patterns |
+| [SDK MCP Mapping](./sdk-mcp-mapping.md) | How MCP tools map to SDK and API operations |
+| [Event Replay Cookbook](./event-replay-cookbook.md) | Patterns for replaying historical events |
+| [Getting Started Guide](./getting-started.md) | Step-by-step setup from zero to first event |
+| [Quick Reference Card](./quick-reference.md) | Single-page cheat sheet for common SDK operations |

package/docs/sdk-mcp-mapping.md

@@ -0,0 +1,20 @@
+# MCP tools vs SDK (high level)
+
+This is a **guide for agents**, not an exhaustive OpenAPI listing. MCP stays on
+**HTTP**; the SDK adds **typed REST** and the **stream data plane** for live
+I/O.
+
+| Area                     | MCP (typical)                             | SDK                                                                      |
+| ------------------------ | ----------------------------------------- | ------------------------------------------------------------------------ |
+| Catalog / discovery      | `POST /ai/mcp/tools/call` discovery tools | `client.discovery.*`, `client.catalog.search`                            |
+| Data products            | MCP dataproduct tools                     | `client.data_products` (get, list, create, query, stream, replay, …)     |
+| Flows / workflows        | MCP / Studio workflows                    | `client.flows`, `client.workflows`, `client.projects`                    |
+| Connections / connectors | MCP connectors                            | `client.connections`, `client.connectors`                                |
+| Org / instances          | MCP org / instances                       | `client.instances.list()`, config `instance_id`                          |
+| Live queue I/O           | Not a substitute for the stream runtime   | `client.queues`, `client.flows.get_writer`, `data_products.stream` + bus |
+
+When unsure: **MCP for provisioning and agent tool calls**; **SDK (+ stream bus
+config) for runtime** services running in your VPC or CI.
+
+Skills in **loxtep-plugins-skills** (`loxtep-sdk` skill) duplicate bootstrap
+snippets so MCP servers do not need codegen tools.

package/docs/sdk-pairing.md

@@ -0,0 +1,42 @@
+# MCP, CLI, and Node SDK — one pairing story
+
+## Same API host
+
+Use one base URL everywhere:
+
+- **REST / SDK:** `LOXTEP_API_URL` or `config.json` → `api_url` (SDK
+  `LoxtepClient({ api_url })`).
+- **Customer MCP / browser login:** optional `api_base_url` in
+  `~/.loxtep/credentials.json` (same value; trailing slash optional).
+
+The CLI and SDK resolve tokens in this order: **`LOXTEP_AUTH_TOKEN`** →
+**`~/.loxtep/credentials.json`** (shared: `loxtep login` and
+`npx @loxtep/customer-mcp-server login`). You should not need two logins for the
+same machine if you use one file-based flow.
+
+## When to use what
+
+| Job                                                                         | Tool                                                           |
+| --------------------------------------------------------------------------- | -------------------------------------------------------------- |
+| Provision org resources, run catalog/dataproduct MCP tools, IDE agent calls | **Customer MCP** (HTTP tools against the platform)             |
+| Typed REST from Node services, scripts, CI                                  | **`@loxtep/sdk`** (`LoxtepClient`)                             |
+| Quick operator commands, token bootstrap                                    | **`loxtep` CLI** (ships with the SDK)                          |
+| **Live** queue produce/consume (Loxtep streams)                             | **SDK** with stream bus config + AWS principal (not JWT alone) |
+| **Historical** replay / trace-style reads where the platform stores history | **SDK** `data_products.replay()` (control-plane HTTP)          |
+
+There is **no** required joint npm install of MCP and SDK; pair them by
+**config + docs**, not a metapackage.
+
+## Install
+
+```bash
+npm install @loxtep/sdk
+# Node.js 22+ recommended (see package engines).
+```
+
+## See also
+
+- [Control vs data plane](./sdk-control-vs-data-plane.md)
+- [REST SigV4 + JWT vs bus](./sdk-rest-vs-bus.md)
+- [MCP → SDK mapping](./sdk-mcp-mapping.md)
+- [Bus RBAC threat model](./sdk-bus-rbac-threat-model.md)

package/docs/sdk-rest-vs-bus.md

@@ -0,0 +1,131 @@
+# REST (SigV4 + JWT) vs Loxtep stream bus
+
+## Chosen config surfaces (SDK)
+
+The Node SDK uses **(B) persisted `LoxtepConfig` + `loadConfig` / CLI** and
+**(C) an optional second JSON file** referenced by
+`LOXTEP_RSTREAMS_CONFIG_FILE`, merged **on top of** the `streams` key from the
+main config file (same PascalCase `ConfigurationResources` shape as in code; see
+`parseStreamsPartial`).
+
+- **Main file** (`~/.loxtep/config.json` or path you pass to `loadConfig`):
+  `api_url`, `organization_id`, `project_id`, `instance_id`, `region`, optional
+  `streams`.
+- **Optional bus-only file**
+  (`LOXTEP_RSTREAMS_CONFIG_FILE=/path/to/rstreams.json`): same JSON shape as the
+  `streams` object; **wins** over overlapping keys from the main file. Useful
+  for copying managed infra JSON without mixing it with `api_url`.
+- **Per-key env** `LEO_*` / `AWS_REGION` (and friends) are still merged inside
+  **`resolveStreamsConfiguration`** when the client is constructed — see
+  [`configuration.ts`](../src/rstreams/configuration.ts).
+- **Observe-proxied stream config:** `client.observe.stream_config()` calls
+  **`GET /observe/stream-config`** on the **app** microservice (JWT + SigV4 like
+  other `/observe/*` routes). The control plane **proxies** to the instance’s
+  botmon **`GET /botmon/api/stream_config`**, which returns allowlisted resource
+  names (same shape as `streams` / `resolveStreamsConfiguration`). Instance
+  records should store **`connection_details.observe_api.rstreams_secret_arn`**
+  (full Secrets Manager ARN) for provisioning reference; the upstream lambda may
+  also read from its deployed `leosdk` env.
+- Instances using Observe transport mode **`direct`** cannot use this proxy path
+  today (the proxy returns **409**); use local `streams` / `LEO_*` or call the
+  instance observe API directly with your stack’s auth model.
+- You must still align bus settings with the **same** Loxtep instance as your
+  workload (see [Instance and bus alignment](#instance-and-bus-alignment)).
+
+`LOXTEP_REGION` overrides `region` from the main file (HTTP SigV4 to API
+Gateway). **Do not** put long-lived AWS access keys or JWT refresh material into
+these JSON files.
+
+## REST
+
+- Every SDK HTTP call is **AWS SigV4** signed and carries **`x-jwt-token`** when
+  `get_token` / `auth` supplies a JWT.
+- In **Node**, if you do not pass `credentials` and have no ambient AWS chain,
+  signing fails with `AWS credentials not available`.
+- **CLI** uses a fixed dummy `accessKeyId` / `secretAccessKey` pair where the
+  gateway accepts it; **library users** must supply real IAM/user credentials or
+  the same dummy pattern only in dev stacks that explicitly allow it.
+- **Never** paste IAM secrets into MCP token files or JWT-only env vars — keep
+  API JWT and bus IAM in separate mechanisms.
+
+## Stream bus (data plane)
+
+- Configured via `LoxtepClient({ streams: { ... } })`, **`LEO_*` env**, merged
+  config + optional `LOXTEP_RSTREAMS_CONFIG_FILE`, and/or **CLI** loading all of
+  the above into the client. Merge order for file-backed partials: **main config
+  `streams`**, then **optional rstreams file**; **constructor `streams`** and
+  `resolveStreamsConfiguration` also fold in `process.env`.
+- **Tests** may inject `streams_sdk` to bypass real AWS / runtime construction.
+
+## File permissions (bus JSON)
+
+Treat files that contain `LeoEvent`, `LeoStream`, Kinesis, S3, etc. as
+**sensitive infrastructure identifiers** (not passwords, but still
+opsec-relevant). Prefer:
+
+- `chmod 600` (user read/write only) for `rstreams.json` and restrictive
+  permissions on `~/.loxtep/`.
+- CI secrets or a server API that RBAC-gates bus metadata — avoid committing
+  these values to public repos.
+
+## Security model (what goes where)
+
+| Item                                                                               | Treat as                                                 | Recommended handling                                                                                                                                     |
+| ---------------------------------------------------------------------------------- | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `LeoEvent`, `LeoStream`, Kinesis, S3, Firehose, `LeoSettings`, `LeoCron`, `Region` | **Sensitive infrastructure identifiers** (not passwords) | Config file with restrictive permissions, env vars in CI, or a server API that RBAC-gates; avoid public repos.                                           |
+| AWS access keys / STS session                                                      | **Secrets**                                              | Use HTTP client [`credentials`](../src/client/types.ts) and/or `refresh` **`aws_credentials`**; do not put into `~/.loxtep/config.json`.                 |
+| `rstreams/auth` / Leo auth                                                         | **Secrets / separate concern**                           | Do not copy bot auth material into the SDK config file; only wire **resources** the way `RStreamsSdk` / `leo-sdk` expect for the **data plane** you use. |
+
+`leo-sdk` uses the **default AWS credential provider chain** in the process for
+DynamoDB/Kinesis; the SDK’s `credentials` option is for **SigV4 to the Loxtep
+HTTP API**, not automatically for the stream runtime unless the same process
+exports profiles/env so both paths see the right IAM **in the bus
+account/region**.
+
+## Instance and bus alignment
+
+- **`LOXTEP_INSTANCE_ID` / `LoxtepConfig.instance_id`** scope **org/instance
+  HTTP** calls. They **do not** auto-select or load `LEO_*` / `streams` for that
+  instance.
+- The physical bus (DynamoDB table names, Kinesis, S3) is **1:1 with a Loxtep
+  instance deployment** in AWS. You must pass **`ConfigurationResources` that
+  match the instance** whose bus hosts the queues you use — wrong instance ⇒
+  wrong table names, silent misrouting, or hard failures.
+- Until a **platform** “bus metadata for `instance_id`” API exists, align bus
+  config via **per-profile** files, **env**, or **separate** `rstreams.json` per
+  instance you work with.
+
+## RBAC vs IAM (read vs write on bus)
+
+**Principle:** the **API JWT** and Loxtep **RBAC** apply to **SigV4 REST** only.
+The **stream data plane** (Leo) uses **AWS credentials** to DDB/Kinesis/S3 for
+the configured `ConfigurationResources` — **not** the JWT. See
+[sdk-bus-rbac-threat-model.md](./sdk-bus-rbac-threat-model.md) for the full
+threat discussion.
+
+| Path                                            | Write vs read (Leo) | RBAC (Loxtep)? | IAM (AWS)?                                              |
+| ----------------------------------------------- | ------------------- | -------------- | ------------------------------------------------------- |
+| **Bus: `putEvents`**                            | Write               | No             | Yes                                                     |
+| **Bus: `offloadEvents`** (tail / `open_reader`) | Read                | No             | Yes (read may differ from write policy)                 |
+| **HTTP: `data_products.stream` / replay**       | “Read” over REST    | **Yes**        | SigV4 + JWT; bus tables not used by client in that mode |
+| **HTTP: `get_queue_metadata`, checkpoints**     | Metadata            | **Yes**        | SigV4 for those endpoints                               |
+
+**Cognito / session alignment:** when the server returns **`aws_credentials`**
+on refresh, the Node SDK and CLI can update **HTTP** SigV4 credentials. If the
+**same** session is what the process uses for the **default AWS provider
+chain**, Leo and HTTP share a **common TTL**; if only the HTTP `credentials`
+object is updated, Leo’s ambient chain may still differ (documented integration
+risk).
+
+## Refresh
+
+- **JWT refresh:** `POST /auth/refresh` with refresh token; CLI wires
+  **TokenManager** + **401 retry** when tokens come from the credentials file
+  (not for `LOXTEP_AUTH_TOKEN`-only mode).
+- When the server session was created from password login or from
+  `POST /auth/cognito-exchange` with `cognito_refresh_token`, the refresh
+  response may include **`aws_credentials`** (same snake_case fields as login).
+  The Node SDK maps these into **`RefreshResponse.aws_credentials`**; the CLI
+  updates its mutable SigV4 credential object after each successful refresh when
+  STS is present. **Legacy sessions** without a stored Cognito refresh omit
+  `aws_credentials` (JWT-only refresh).