@loxtep/sdk 0.1.0

package/docs/getting-started.md

@@ -0,0 +1,332 @@
+# SDK Getting Started Guide
+
+Get from zero to your first event in under 5 minutes. This guide walks you
+through installing the Loxtep SDK, authenticating, and writing your first event
+to a data product — all with a single method call.
+
+## Table of Contents
+
+- [Prerequisites](#prerequisites)
+- [Step 1: Install the SDK](#step-1-install-the-sdk)
+- [Step 2: Log In](#step-2-log-in)
+- [Step 3: Write Your First Event](#step-3-write-your-first-event)
+- [Step 4: Read Events Back](#step-4-read-events-back)
+- [Complete Example](#complete-example)
+- [Troubleshooting](#troubleshooting)
+- [Next Steps](#next-steps)
+
+---
+
+## Prerequisites
+
+Before you begin, make sure you have:
+
+| Requirement | Details |
+|-------------|---------|
+| **Node.js 22+** | Required for the Node.js SDK (`@loxtep/sdk`). Check with `node --version`. |
+| **Loxtep account** | You need an account with the **developer** role. Ask your org admin if you don't have one. |
+| **A deployed data product** | Your organization must have at least one data product deployed to an instance. |
+
+---
+
+## Step 1: Install the SDK
+
+```bash
+npm install @loxtep/sdk
+```
+
+Or with your preferred package manager:
+
+```bash
+pnpm add @loxtep/sdk
+# or
+yarn add @loxtep/sdk
+```
+
+Verify the installation:
+
+```bash
+npx loxtep --version
+```
+
+---
+
+## Step 2: Log In
+
+Authenticate with your Loxtep account. This stores credentials locally so the
+SDK can make API calls on your behalf.
+
+```bash
+npx loxtep login
+```
+
+Follow the prompts to enter your email and password. On success you'll see:
+
+```
+✓ Logged in successfully. Credentials saved to ~/.loxtep/credentials.json
+```
+
+**Alternative: use an environment variable.** If you're running in CI/CD or a
+container, set the token directly instead of using `loxtep login`:
+
+```bash
+export LOXTEP_AUTH_TOKEN="your-jwt-token"
+```
+
+The SDK checks `LOXTEP_AUTH_TOKEN` first, then falls back to
+`~/.loxtep/credentials.json`.
+
+---
+
+## Step 3: Write Your First Event
+
+With the new data-product-centric API, you write events by referencing a data
+product by name. The SDK resolves all the plumbing (queue names, bot IDs, stream
+bus configuration) automatically.
+
+```typescript
+import { LoxtepClient } from '@loxtep/sdk';
+
+const client = new LoxtepClient({
+  api_url: process.env.LOXTEP_API_URL,
+  auth: { type: 'jwt', token: process.env.LOXTEP_TOKEN },
+});
+
+// Write events to a data product by name
+const writer = await client.data_products.get_writer('my-data-product');
+writer.write({ id: '123', name: 'Alice' });
+await writer.close();
+```
+
+That's it. No workflow IDs, no deployment lookups, no queue name conventions.
+The SDK resolves the data product's runtime bindings, connects to the correct
+stream bus, and writes directly to the data product queue.
+
+### What happens under the hood
+
+1. The SDK looks up the data product by name via the platform API
+2. It reads the `deployment_bindings` to find the queue name, bot ID, and instance
+3. It resolves the stream bus configuration from the instance record
+4. It initializes the stream SDK and returns a writer pointed at the correct queue
+
+### Writer options
+
+You can customize the writer behavior with an options object:
+
+```typescript
+const writer = await client.data_products.get_writer('my-data-product', {
+  bot_id: 'custom-bot-id',   // Override the resolved bot identity
+  batch_size: 500,            // Events per batch (default: 100)
+  max_retries: 5,             // Retry attempts on failure (default: 3)
+});
+```
+
+---
+
+## Step 4: Read Events Back
+
+Reading from a data product is just as simple:
+
+```typescript
+const reader = await client.data_products.get_reader('my-data-product');
+
+for await (const event of reader) {
+  console.log('Received:', event);
+}
+```
+
+The reader yields events as an async iterable. By default it generates a
+reader bot ID of `sdk-reader-{data-product-name}` for checkpoint tracking.
+
+### Reader options
+
+```typescript
+const reader = await client.data_products.get_reader('my-data-product', {
+  bot_id: 'my-consumer',     // Custom reader identity for checkpointing
+  from: 'z/2024/01/01',      // Start position (default: latest checkpoint)
+  batch_size: 200,            // Events per batch (default: 100)
+});
+```
+
+---
+
+## Complete Example
+
+```typescript
+// first-event.ts
+// Run: npx tsx first-event.ts
+
+import { LoxtepClient } from '@loxtep/sdk';
+
+async function main() {
+  const client = new LoxtepClient({
+    api_url: process.env.LOXTEP_API_URL,
+    auth: { type: 'jwt', token: process.env.LOXTEP_TOKEN },
+  });
+
+  // Write an event to a data product
+  const writer = await client.data_products.get_writer('my-data-product');
+
+  writer.write({
+    id: `event-${Date.now()}`,
+    timestamp: new Date().toISOString(),
+    payload: { message: 'Hello from the Loxtep SDK!' },
+  });
+
+  await writer.close();
+  console.log('✓ Event written');
+
+  // Read it back
+  const reader = await client.data_products.get_reader('my-data-product', {
+    bot_id: 'quickstart-reader',
+  });
+
+  for await (const event of reader) {
+    console.log('✓ Event received:', JSON.stringify(event, null, 2));
+    break; // Just read the first one to verify
+  }
+
+  console.log('Done!');
+}
+
+main().catch(console.error);
+```
+
+---
+
+## Troubleshooting
+
+### Data product not found
+
+**Symptom:** `NotFoundError: Data product 'my-data-product' not found`
+
+**Cause:** The SDK could not find a data product matching the name you provided.
+
+**Fix:**
+
+1. Verify the data product name is spelled correctly (names are case-sensitive).
+2. Check that the data product's workflow has been deployed to an instance.
+3. If you're using a UUID, confirm it matches an existing data product ID.
+
+```bash
+# List data products in your organization
+npx loxtep data-products list
+```
+
+---
+
+### Data product not deployed
+
+**Symptom:** `StreamingError: Data product 'my-data-product' is not deployed.
+Deploy the workflow first.`
+
+**Cause:** The data product exists in the workflow graph but has not been
+deployed to an instance. Runtime bindings (queue name, bot ID) are only
+available after deployment.
+
+**Fix:**
+
+1. Deploy the workflow containing the data product:
+
+```bash
+npx loxtep deploy <workflow-id>
+```
+
+2. Or deploy via the Loxtep UI: **Workflows** → select your workflow → **Deploy**.
+
+3. If using an AI assistant with MCP, use the `deploy_workflow` tool.
+
+---
+
+### Unable to resolve stream configuration
+
+**Symptom:** `StreamingError: Failed to resolve stream config for instance
+<instance-id>`
+
+**Cause:** The SDK resolved the data product and its instance, but could not
+retrieve the stream bus resource names (DynamoDB tables, Kinesis stream, S3
+bucket) from the instance record.
+
+**Fix:**
+
+1. Verify the instance is fully provisioned and running in the Loxtep UI.
+2. Check that your account has `instances:read` permission on the target instance.
+3. If the instance was recently created, wait a few minutes for provisioning to
+   complete — CloudFormation stack outputs may not be available immediately.
+4. Contact your platform admin if the instance shows as provisioned but
+   resolution still fails.
+
+---
+
+### Multiple data products match
+
+**Symptom:** `AmbiguityError: Multiple data products match 'my-data-product'.
+Found matches on instances: [instance-a, instance-b]`
+
+**Cause:** The same data product name exists on multiple instances and the SDK
+cannot determine which one you mean.
+
+**Fix:**
+
+1. Set `instance_id` in the client options to scope resolution to a specific
+   instance:
+
+```typescript
+const client = new LoxtepClient({
+  api_url: process.env.LOXTEP_API_URL,
+  auth: { type: 'jwt', token: process.env.LOXTEP_TOKEN },
+  instance_id: process.env.LOXTEP_INSTANCE_ID,
+});
+```
+
+2. Or use the data product's UUID instead of its name:
+
+```typescript
+const writer = await client.data_products.get_writer('09fa202b-...');
+```
+
+---
+
+### Missing Auth Token
+
+**Symptom:** `AuthenticationError: No authentication token found`
+
+**Cause:** The SDK cannot find a JWT token. Neither `LOXTEP_TOKEN` is set
+nor `~/.loxtep/credentials.json` exists.
+
+**Fix:**
+
+```bash
+# Option 1: Log in interactively
+npx loxtep login
+
+# Option 2: Set the token directly
+export LOXTEP_TOKEN="your-jwt-token"
+```
+
+---
+
+## Next Steps
+
+Now that you've written and read your first event, explore these resources:
+
+| Resource | Description |
+|----------|-------------|
+| [Architecture Guide](./sdk-ingestion-export-guide.md) | Deep dive into how SDK ingestion and export works under the hood |
+| [Quick Reference Card](./quick-reference.md) | Single-page cheat sheet for common SDK operations |
+| [SDK REST vs Bus](./sdk-rest-vs-bus.md) | Understanding control plane vs data plane configuration |
+| [Event Replay Cookbook](./event-replay-cookbook.md) | Patterns for replaying and reprocessing historical events |
+
+### Lower-level API
+
+If you need explicit control over bot IDs, queue names, or stream configuration,
+the `flows.get_writer` API is still available as a lower-level escape hatch:
+
+```typescript
+const writer = await client.flows.get_writer('<flow_id>', {
+  bot_id: 'my-custom-bot',
+  output_queue_name: 'specific-queue-name',
+});
+```
+
+See the [SDK README](../README.md) for the full API surface.

package/docs/quick-reference.md

@@ -0,0 +1,256 @@
+# SDK Quick Reference Card
+
+Concise cheat sheet for common Loxtep SDK operations. For full walkthroughs,
+see the [Getting Started Guide](./getting-started.md) and the
+[Architecture Guide](./sdk-ingestion-export-guide.md).
+
+---
+
+## Client Initialization
+
+### Node.js
+
+```typescript
+import { LoxtepClient } from '@loxtep/sdk';
+
+const client = new LoxtepClient({
+  api_url: process.env.LOXTEP_API_URL,
+  organization_id: process.env.LOXTEP_ORGANIZATION_ID,
+  project_id: process.env.LOXTEP_PROJECT_ID,
+  instance_id: process.env.LOXTEP_INSTANCE_ID,
+  region: process.env.LOXTEP_REGION,
+});
+```
+
+### Python
+
+```python
+import os
+from loxtep import LoxtepClient
+
+client = LoxtepClient(
+    api_url=os.environ["LOXTEP_API_URL"],
+    organization_id=os.environ["LOXTEP_ORGANIZATION_ID"],
+    project_id=os.environ["LOXTEP_PROJECT_ID"],
+    instance_id=os.environ["LOXTEP_INSTANCE_ID"],
+    region=os.environ["LOXTEP_REGION"],
+)
+```
+
+> If you ran `loxtep init`, the SDK reads defaults from
+> `~/.loxtep/config.json` — you can use `new LoxtepClient()` with no args.
+
+---
+
+## Writing Events
+
+### Node.js
+
+```typescript
+const writer = await client.flows.get_writer('<flow_id>', {
+  bot_id: 'my-bot',
+  output_queue_name: 'raw-events',
+});
+
+writer.write({ id: 'evt-1', payload: { key: 'value' } });
+writer.write({ id: 'evt-2', payload: { key: 'value' } });
+
+await writer.close(); // flushes all buffered events
+```
+
+### Python
+
+```python
+writer = client.flows.get_writer("<flow_id>", {
+    "bot_id": "my-bot",
+    "output_queue_name": "raw-events",
+})
+
+writer.write({"id": "evt-1", "payload": {"key": "value"}})
+writer.write({"id": "evt-2", "payload": {"key": "value"}})
+
+writer.close()  # flushes all buffered events
+```
+
+---
+
+## Reading Events
+
+### Node.js
+
+```typescript
+const reader = await client.queues.open_reader({
+  bot_id: 'my-reader',
+  queue_name: 'raw-events',
+});
+
+for await (const event of reader.read()) {
+  console.log(event);
+}
+
+reader.close();
+```
+
+### Python
+
+```python
+reader = client.queues.open_reader({
+    "bot_id": "my-reader",
+    "queue_name": "raw-events",
+})
+
+for event in reader.read():
+    print(event)
+
+reader.close()
+```
+
+---
+
+## Data Products
+
+### Stream Live Events
+
+#### Node.js
+
+```typescript
+const stream = await client.data_products.stream('<data_product_id>', {
+  bot_id: 'my-stream-bot',
+});
+
+for await (const event of stream) {
+  console.log(event);
+}
+```
+
+#### Python
+
+```python
+stream = client.data_products.stream("<data_product_id>", {
+    "bot_id": "my-stream-bot",
+})
+
+for event in stream:
+    print(event)
+```
+
+### Replay Historical Events
+
+#### Node.js
+
+```typescript
+const events = await client.data_products.replay('<data_product_id>', {
+  start: '2024-01-01T00:00:00Z',
+  end: '2024-01-02T00:00:00Z',
+});
+```
+
+#### Python
+
+```python
+events = client.data_products.replay("<data_product_id>", {
+    "start": "2024-01-01T00:00:00Z",
+    "end": "2024-01-02T00:00:00Z",
+})
+```
+
+### List and Get Data Products
+
+#### Node.js
+
+```typescript
+const products = await client.data_products.list();
+const product = await client.data_products.get('<data_product_id>');
+```
+
+#### Python
+
+```python
+products = client.data_products.list()
+product = client.data_products.get("<data_product_id>")
+```
+
+---
+
+## Flows and Connectors
+
+### List Flows
+
+#### Node.js
+
+```typescript
+const flows = await client.flows.list();
+```
+
+#### Python
+
+```python
+flows = client.flows.list()
+```
+
+### Get Connector
+
+#### Node.js
+
+```typescript
+const connector = await client.connectors.get('<connector_id>');
+```
+
+#### Python
+
+```python
+connector = client.connectors.get("<connector_id>")
+```
+
+---
+
+## Quick Reference Table
+
+| Task | Node.js | Python |
+|------|---------|--------|
+| **Init client** | `new LoxtepClient(options)` | `LoxtepClient(**options)` |
+| **Write events** | `client.flows.get_writer(flow_id, opts)` → `writer.write(evt)` → `writer.close()` | Same API |
+| **Read events** | `client.queues.open_reader({ bot_id, queue_name })` → `reader.read()` → `reader.close()` | Same API |
+| **Stream live** | `client.data_products.stream(id, opts)` | Same API |
+| **Replay history** | `client.data_products.replay(id, opts)` | Same API |
+| **List data products** | `client.data_products.list()` | Same API |
+| **Get data product** | `client.data_products.get(id)` | Same API |
+| **List flows** | `client.flows.list()` | Same API |
+| **Get connector** | `client.connectors.get(id)` | Same API |
+
+---
+
+## CLI Shortcuts
+
+```bash
+# Log in
+npx loxtep login
+
+# Initialize config
+npx loxtep init
+
+# Export connector config (shell, json, or env format)
+npx loxtep config export --from-connector "<connector_id>" --format sh
+npx loxtep config export --from-connector "<connector_id>" --format json
+npx loxtep config export --from-connector "<connector_id>" --format env
+```
+
+---
+
+## Auth Precedence
+
+The SDK resolves authentication in this order:
+
+1. `LOXTEP_AUTH_TOKEN` environment variable
+2. `~/.loxtep/credentials.json` (from `loxtep login`)
+
+---
+
+## Further Reading
+
+| Resource | Description |
+|----------|-------------|
+| [Getting Started Guide](./getting-started.md) | Zero-to-first-event walkthrough |
+| [Architecture Guide](./sdk-ingestion-export-guide.md) | Control plane vs data plane, data flow diagrams |
+| [SDK REST vs Bus](./sdk-rest-vs-bus.md) | Configuration surfaces and credential handling |
+| [Event Replay Cookbook](./event-replay-cookbook.md) | Patterns for replaying historical events |

package/docs/sdk-bus-rbac-threat-model.md

@@ -0,0 +1,46 @@
+# Bus access — threat model and direction
+
+## Assets
+
+- **API JWT:** org-scoped, audited via normal app auth; used only on **SigV4
+  REST**.
+- **Loxtep stream bus:** Kinesis/Dynamo/S3 tables and streams from your
+  **instance / stack** env; accessed with the **stream runtime** and **AWS
+  credentials** (not JWT alone).
+
+## Threats
+
+1. **Confused deputy:** treating a user JWT as proof of bus access would let any
+   token holder touch queues if they guessed names.
+2. **Over-privileged CI:** long-lived IAM keys on developer laptops co-located
+   with refresh tokens double the blast radius.
+3. **Silent god credentials:** implicit `~/.aws/credentials` for the bus account
+   without org policy mapping.
+
+## Mitigations (target)
+
+- **Separate issuance:** bus-capable material is issued only through **platform
+  APIs** evaluated with the **same RBAC** as REST (resource + action), with
+  **short TTL** and **scoped** policy (queue/bot prefixes).
+- **Revocation:** disabling org role or bus grant immediately stops new
+  issuance; in-flight TTL bounded.
+- **Audit:** who received bus credentials, when, for which instance/data
+  product.
+- **No hand-rolled per-customer IAM** in app code; use STS assume-role or a
+  documented `bus session` API when shipped.
+
+## Current state (honest)
+
+- SDK expects operators to configure the **stream data plane** + **AWS** per
+  environment, or to call **`GET /observe/stream-config`** (same Observe
+  **proxy** path as `GET /observe/sdk-config`) with **`instances:read`** to
+  receive allowlisted **Leo/RStreams resource identifiers** resolved in the
+  **instance account** (not a JWT substitute for IAM on the bus).
+- **`loxtep bus login`** is a **placeholder** until a platform **bus session**
+  (or STS exchange) endpoint ships; see CLI help output.
+
+## ADR stub
+
+**Decision (interim):** ship the Loxtep **stream** data plane in the SDK with
+env/stack-based bus config; document JWT vs bus split; add RBAC-gated issuance
+in a follow-up release when the API exists.

package/docs/sdk-control-vs-data-plane.md

@@ -0,0 +1,32 @@
+# Control plane vs data plane (SDK)
+
+## Control plane (REST)
+
+- **SigV4-signed HTTP** to the API gateway **plus** `x-jwt-token` (org/user
+  JWT).
+- Used for: catalogs, data product CRUD, flow definitions, deploy, checkpoints
+  **metadata**, `data_products.replay`, query, contracts, MCP-proxy style
+  `discovery` calls, etc.
+- **AWS credentials** are still required for SigV4 unless your environment
+  injects them (CLI uses a documented dummy pair only where the gateway allows
+  it).
+
+## Data plane (Loxtep streams)
+
+- **Live** events: `flows.get_writer().close()`, `queues.open_reader` /
+  `open_writer`, and `data_products.stream` when `bot_id` + configured bus — all
+  go through the Loxtep **stream runtime** (DynamoDB / Kinesis / S3, etc. per
+  your stack) via `LoxtepClientOptions.streams` and instance env, or a prebuild
+  passed as `streams_sdk`.
+- **Not** “JWT ⇒ bus.” Bus access uses **AWS principals** scoped to the stream
+  infrastructure. Platform **RBAC** should gate _who_ may receive short-lived
+  bus material (see
+  [sdk-bus-rbac-threat-model.md](./sdk-bus-rbac-threat-model.md)); today
+  operators wire stack access and env.
+
+## Project vs queue
+
+- **Project** scopes control-plane APIs: list flows, deploy workflow graphs,
+  resolve ingestion queue names from flow metadata.
+- **Queue** is the runtime unit on the bus: readers/writers are always
+  `bot_id` + `queue_name` on the stream data plane.