@hotmeshio/long-tail 0.1.14 → 0.1.16

package/docs/compiler.md

@@ -0,0 +1,311 @@
+# The Workflow Compiler
+
+You wrote a durable workflow. It works. `proxyActivities`, `sleep`, `condition` — the Temporal-like API is productive and familiar. But under the hood, the durable engine replays the entire workflow function on every wake-up. Sleep three times in a ten-step workflow? Steps 1–3 replay on wake one. Steps 1–6 replay on wake two. Steps 1–9 replay on wake three. The replay is deterministic — it skips completed activities — but the function still executes from the top every time.
+
+The compiled YAML DAG does the same work without replay. Each step fires exactly once. State flows explicitly between activities through input mappings. No function re-execution, no replay loop, no wasted cycles.
+
+`ltc` is the compiler. It reads your TypeScript source and produces an equivalent YAML DAG. Write procedural because it's productive. Run the DAG because it's fast.
+
+---
+
+## The Analogy
+
+`tsc` compiles TypeScript to JavaScript. You write in the language that's expressive. You run the artifact that's efficient. The source is authoritative — you edit it, test it, review it. The compiled output is what actually executes.
+
+`ltc` does the same for workflows:
+
+```
+  TypeScript source       ltc compile        YAML DAG
+  (developer-friendly) ──────────────────→ (execution-optimized)
+
+  assembly-line.ts     →   assembly-line.compiled.yaml
+```
+
+The `.ts` file is the spec. The `.compiled.yaml` is the optimized execution. Both live in the repo. A reviewer reads both side-by-side and traces every line of orchestration logic to its DAG equivalent.
+
+---
+
+## Quick Start
+
+```bash
+npm install -g @hotmeshio/long-tail
+export ANTHROPIC_API_KEY=sk-ant-...
+
+ltc compile workflows/basic-echo/index.ts
+```
+
+Output:
+
+```
+  ✓ index.ts → workflows/basic-echo/index.compiled.yaml
+    1 activities · 2 inputs (message, sleepSeconds) · topic: basic.echo
+
+  Compiled 1 workflow in 18.1s
+```
+
+The compiled YAML appears adjacent to the source file.
+
+---
+
+## Before and After
+
+### The source (procedural)
+
+```typescript
+import { Durable } from '@hotmeshio/hotmesh';
+import * as activities from './activities';
+
+const { echo } = Durable.workflow.proxyActivities<typeof activities>({
+  activities,
+});
+
+export async function basicEcho(envelope: LTEnvelope): Promise<any> {
+  const { message = 'Hello, Long Tail!', sleepSeconds = 1 } = envelope.data;
+
+  // 1. Durable sleep — the engine replays to this point on wake-up
+  await Durable.workflow.sleep(`${sleepSeconds} seconds`);
+
+  // 2. Activity call — replayed as a no-op if already completed
+  const echoResult = await echo({ message });
+
+  return {
+    type: 'return' as const,
+    data: { ...echoResult, sleepSeconds, userId: envelope.lt?.userId },
+  };
+}
+```
+
+This function executes three times: once on initial invocation, once when the sleep timer fires, and once when the echo activity completes. Each time, the engine replays from the top, skipping completed steps. For a two-step workflow this is fine. For a twenty-step workflow with multiple sleeps, the replay cost compounds.
+
+### The compiled output (DAG)
+
+```yaml
+app:
+  id: longtail
+  version: '1'
+  graphs:
+    - subscribes: basic.echo
+      expire: 300
+      input:
+        schema:
+          type: object
+          properties:
+            message:
+              type: string
+              default: 'Hello, Long Tail!'
+            sleepSeconds:
+              type: number
+              default: 1
+      activities:
+        trigger_m7qz:
+          type: trigger
+          output:
+            schema:
+              type: object
+              properties:
+                message: { type: string }
+                sleepSeconds: { type: number }
+
+        delay_m7qz:
+          type: hook
+          sleep: '{trigger_m7qz.output.data.sleepSeconds}'
+
+        echo_m7qz:
+          type: worker
+          topic: basic.echo
+          input:
+            maps:
+              message: '{trigger_m7qz.output.data.message}'
+              workflowName: echo
+
+      transitions:
+        trigger_m7qz:
+          - to: delay_m7qz
+        delay_m7qz:
+          - to: echo_m7qz
+```
+
+### The mapping
+
+| TypeScript | YAML | What happens |
+|---|---|---|
+| `envelope.data` destructuring | `trigger` activity with input schema | Inputs declared as typed schema with defaults |
+| `Durable.workflow.sleep(N)` | `hook` activity with `sleep` field | Engine sets a timer; no replay on wake |
+| `await echo({ message })` | `worker` activity with input `maps` | Worker fires once; result stored in job hash |
+| `return { data: ... }` | `job.maps` on the final activity | Output fields plucked from activity results |
+
+Each step executes exactly once. The `sleep` hook pauses the DAG. When the timer fires, execution resumes at `echo_m7qz` — not from the top of the function.
+
+---
+
+## Discovery
+
+Point `ltc` at a directory and it finds workflow files automatically:
+
+```bash
+ltc compile examples/workflows/ --dry-run
+```
+
+```
+  Found 10 workflows:
+
+  ● assembly-line/index.ts
+    Function: assemblyLine  ·  HotMesh Durable
+    Control flow: startChild, condition
+
+  ● basic-echo/index.ts
+    Function: basicEcho  ·  HotMesh Durable
+    Activities: echo
+    Control flow: sleep
+
+  ● basic-signal/index.ts
+    Function: basicSignal  ·  HotMesh Durable
+    Activities: ltCreateEscalation, processApproval
+    Control flow: conditionLT
+
+  ● kitchen-sink/index.ts
+    Function: kitchenSink  ·  HotMesh Durable
+    Activities: greet, fetchData, transformData, notifyComplete
+    Control flow: sleep, executeLT
+```
+
+### What the scanner looks for
+
+| Pattern | Classification |
+|---|---|
+| `proxyActivities` / `Durable.workflow.proxyActivities` | HotMesh Durable workflow |
+| `@temporalio/workflow` import | Temporal workflow |
+| `sleep`, `condition`, `signal` | Control flow primitives |
+| `startChild`, `executeChild`, `executeLT` | Composition (parent workflow) |
+| Exported async function | Workflow entry point |
+
+Files in `node_modules/`, `build/`, `dist/`, and test files (`*.test.ts`, `*.spec.ts`) are excluded.
+
+---
+
+## How Compilation Works
+
+The compiler is not an AST parser. It uses the LLM to translate orchestration logic, grounded by structural metadata extracted from the source.
+
+1. **Read source** — file or inline string
+2. **Extract metadata** — lightweight regex extraction: activity names, durable primitives, envelope fields, import paths, control flow markers (loops, Promise.all, conditionals, escalation)
+3. **Resolve activities** — if the source imports activity modules, the compiler reads them too and includes them in the LLM context
+4. **LLM translation** — the source code, extracted metadata, and the full HotMesh YAML specification are sent to the LLM. Temperature 0. Up to 3 retry attempts on parse failure.
+5. **Fix patterns** — known LLM anti-patterns in YAML pipe expressions are corrected deterministically
+
+The metadata extraction is intentionally lightweight — regex, not a TypeScript AST. It gives the LLM structural hints (which functions are activities, which primitives are used) without requiring `ts-morph` or the TypeScript compiler API as a dependency.
+
+---
+
+## Composition
+
+When a workflow uses `startChild` or `executeChild`, the compiler resolves the child workflow source through imports and compiles children first (leaf-first ordering). The parent's compiled YAML references each child by topic using the `await` activity type — the same mechanism used by the Pipeline Designer's Plan Build mode.
+
+```
+parent.ts  ──→  parent.compiled.yaml
+  └─ startChild(worker)
+       └─ worker.ts  ──→  worker.compiled.yaml
+```
+
+All compiled workflows in the same composition share the same `app.id` namespace, enabling cross-graph invocation.
+
+---
+
+## Three Surfaces
+
+The compiler is the same function regardless of how you invoke it.
+
+### CLI
+
+```bash
+ltc compile workflows/assembly-line.ts
+ltc compile src/workflows/ --dry-run
+ltc compile --model claude-sonnet-4-6
+```
+
+### HTTP API
+
+```bash
+curl -X POST http://localhost:3000/api/yaml-workflows/from-durable \
+  -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "source": "export async function basicEcho(envelope) { ... }",
+    "workflow_name": "basicEcho",
+    "name": "basic_echo"
+  }'
+```
+
+### SDK
+
+```typescript
+import { createClient } from '@hotmeshio/long-tail/sdk';
+
+const lt = createClient({ auth: { userId: 'system' } });
+
+const result = await lt.yamlWorkflows.compileDurable({
+  source: fs.readFileSync('workflows/assembly-line.ts', 'utf-8'),
+  workflow_name: 'assemblyLine',
+  name: 'assembly_line',
+});
+```
+
+The CLI writes `.compiled.yaml` files to disk. The HTTP and SDK surfaces store the compiled workflow in the database and return the record. All three use the same underlying `compileDurableToYaml()` function.
+
+---
+
+## Configuration
+
+| Variable | Default | Description |
+|---|---|---|
+| `ANTHROPIC_API_KEY` | — | Required for Claude models |
+| `OPENAI_API_KEY` | — | Required for OpenAI models |
+| `LT_LLM_MODEL_PRIMARY` | `claude-sonnet-4-6` | Model used for compilation |
+| `LT_LLM_BASE_URL` | — | Custom endpoint for OpenAI-compatible models |
+
+The CLI loads `.env` automatically if present in the current directory.
+
+---
+
+## CLI Reference
+
+```
+Usage: ltc compile [options] [target]
+
+Compile durable TypeScript workflows to YAML DAGs
+
+Arguments:
+  target                  File or directory to compile (default: current directory)
+
+Options:
+  --dry-run               Show discovered workflows without compiling
+  -o, --output <dir>      Output directory (default: adjacent to source file)
+  --model <model>         LLM model to use (default: claude-sonnet-4-6)
+  --function <name>       Workflow function name (auto-detected if omitted)
+  -h, --help              Display help
+```
+
+### Examples
+
+```bash
+# Compile a single file
+ltc compile workflows/basic-echo/index.ts
+
+# Scan and compile all workflows in a directory
+ltc compile src/workflows/
+
+# Compile everything in the current directory
+ltc compile
+
+# Preview what would be compiled
+ltc compile examples/workflows/ --dry-run
+
+# Use a specific model
+ltc compile workflows/complex.ts --model claude-opus-4-6
+
+# Write output to a separate directory
+ltc compile workflows/ -o compiled/
+
+# Compile a specific function from a multi-export file
+ltc compile workflows/multi.ts --function assemblyLine
+```

package/docs/events.md

@@ -26,6 +26,25 @@ await start({
 
 `start()` handles adapter connection and graceful disconnection on shutdown automatically.
 
+## Dashboard Transport Selection
+
+The dashboard auto-detects its event transport via `GET /api/settings`. By default, Socket.IO is reported — it works in-process with no additional infrastructure.
+
+For multi-container deployments where the API and workers run as separate processes, set `EVENT_TRANSPORT=nats` to tell the dashboard to connect via NATS WebSocket instead of Socket.IO:
+
+```yaml
+# docker-compose.yml
+services:
+  app:
+    environment:
+      - EVENT_TRANSPORT=nats
+      - NATS_URL=nats://nats:4222
+      - NATS_WS_URL=ws://localhost:9222
+      - NATS_TOKEN=your-token
+```
+
+Both adapters still publish events regardless of `EVENT_TRANSPORT` — the setting only controls what the dashboard listens on. This means server-side event consumers (callbacks, NATS subscribers) work independently of the dashboard transport.
+
 ## LTEvent
 
 Every published event conforms to the `LTEvent` interface:

package/docs/workflows.md

@@ -693,3 +693,22 @@ OPENAI_API_KEY=sk-... npm run test:vision
 # Full backend suite
 npm test
 ```
+
+---
+
+## From Durable to DAG
+
+Every durable workflow you write is a candidate for compilation. The `ltc` CLI reads your workflow source and produces an equivalent YAML DAG that runs without replay overhead.
+
+```bash
+ltc compile workflows/my-workflow.ts
+```
+
+The durable code is the spec — developer-friendly, familiar, testable with standard tools. The compiled YAML is the optimized execution path — each step fires exactly once, state is plucked explicitly from upstream activities, no replay loop.
+
+```
+  my-workflow.ts              →  my-workflow.compiled.yaml
+  (write and test here)           (deploy and run this)
+```
+
+Write procedural because it's productive. Compile because it's fast. See the [Compiler Guide](compiler.md) for details.

package/package.json

@@ -1,9 +1,12 @@
 {
   "name": "@hotmeshio/long-tail",
-  "version": "0.1.14",
+  "version": "0.1.16",
   "description": "Long Tail Workflows — Durable AI workflows with human-in-the-loop escalation. Powered by PostgreSQL.",
   "main": "./build/index.js",
   "types": "./build/index.d.ts",
+  "bin": {
+    "ltc": "./build/bin/ltc.js"
+  },
   "homepage": "https://github.com/hotmeshio/long-tail/",
   "publishConfig": {
     "access": "public"
@@ -44,7 +47,11 @@
     "dashboard:build": "cd dashboard && npm run build",
     "dashboard:test": "cd dashboard && npm test",
     "dashboard:test:watch": "cd dashboard && npm run test:watch",
-    "build:all": "npm run build && npm run dashboard:build"
+    "ltc": "tsx bin/ltc.ts",
+    "build:all": "npm run build && npm run dashboard:build",
+    "release:patch": "npm version patch && git push && git push --tags",
+    "release:minor": "npm version minor && git push && git push --tags",
+    "release:major": "npm version major && git push && git push --tags"
   },
   "keywords": [
     "Long Tail Workflows",
@@ -71,11 +78,14 @@
     "ajv": "^8.20.0",
     "arctic": "^3.7.0",
     "bcryptjs": "^2.4.3",
+    "commander": "^14.0.3",
     "express": "^5.1.0",
     "jsonwebtoken": "^9.0.3",
     "nats": "^2.28.0",
     "openai": "^5.9.0",
+    "ora": "^9.4.0",
     "pg": "^8.13.0",
+    "picocolors": "^1.1.1",
     "pino": "^9.6.0",
     "playwright": "^1.58.2",
     "sharp": "^0.34.5",
@@ -93,6 +103,7 @@
     "rimraf": "^4.4.1",
     "ts-node": "^10.9.1",
     "ts-node-dev": "^2.0.0",
+    "tsx": "^4.21.0",
     "typescript": "^5.0.4",
     "vitest": "^3.2.4"
   }

package/build/api/escalations.d.ts

@@ -1,271 +0,0 @@
-import type { LTApiResult, LTApiAuth } from '../types/sdk';
-/**
- * Create a standalone escalation (not tied to a workflow).
- *
- * Useful for manual work items, support tickets, or approval requests
- * that originate outside the durable workflow engine. The caller must
- * hold the target role or be a superadmin.
- *
- * @param input.type — escalation category (e.g. `"support"`, `"approval"`)
- * @param input.subtype — subcategory for finer routing
- * @param input.role — role responsible for resolving this escalation
- * @param input.description — human-readable summary
- * @param input.priority — 1 (critical) through 4 (low), default 2
- * @param input.envelope — serialized context for the resolver
- * @param input.metadata — arbitrary key-value data (e.g. signal_routing)
- * @param input.escalation_payload — serialized payload for the resolver UI
- * @param auth — authenticated user context (must hold target role or be superadmin)
- * @returns `{ status: 201, data: <escalation record> }`
- */
-export declare function createEscalation(input: {
-    type: string;
-    subtype?: string;
-    role: string;
-    description?: string;
-    priority?: number;
-    envelope?: string;
-    metadata?: Record<string, any>;
-    escalation_payload?: string;
-}, auth: LTApiAuth): Promise<LTApiResult>;
-/**
- * List escalations with optional filters.
- *
- * Results are scoped to the authenticated user's roles unless the user
- * is a superadmin (who sees all roles).
- *
- * @param input.status — filter by `pending`, `resolved`, or `cancelled`
- * @param input.role — filter by assigned role
- * @param input.type — filter by workflow type
- * @param input.subtype — filter by subtype
- * @param input.assigned_to — filter by assigned user ID
- * @param input.priority — filter by priority (1–4)
- * @param input.limit — max results (default: 50)
- * @param input.offset — pagination offset
- * @param input.sort_by — column to sort by (e.g. `created_at`, `priority`)
- * @param input.order — `asc` or `desc`
- * @param auth — authenticated user context (required for role scoping)
- * @returns `{ status: 200, data: { escalations, total } }`
- */
-export declare function listEscalations(input: {
-    status?: string;
-    role?: string;
-    type?: string;
-    subtype?: string;
-    assigned_to?: string;
-    priority?: number;
-    limit?: number;
-    offset?: number;
-    sort_by?: string;
-    order?: string;
-}, auth: LTApiAuth): Promise<LTApiResult>;
-/**
- * List escalations available for claim (pending and not actively claimed).
- *
- * Similar to `listEscalations` but excludes escalations with active claims.
- * Scoped to the authenticated user's roles.
- *
- * @param input.role — filter by role
- * @param input.type — filter by workflow type
- * @param input.subtype — filter by subtype
- * @param input.priority — filter by priority (1–4)
- * @param input.limit — max results (default: 50)
- * @param input.offset — pagination offset
- * @param input.sort_by — column to sort by
- * @param input.order — `asc` or `desc`
- * @param auth — authenticated user context
- * @returns `{ status: 200, data: { escalations, total } }`
- */
-export declare function listAvailableEscalations(input: {
-    role?: string;
-    type?: string;
-    subtype?: string;
-    priority?: number;
-    limit?: number;
-    offset?: number;
-    sort_by?: string;
-    order?: string;
-}, auth: LTApiAuth): Promise<LTApiResult>;
-/**
- * List all distinct escalation type values.
- *
- * @returns `{ status: 200, data: { types: string[] } }`
- */
-export declare function listDistinctTypes(): Promise<LTApiResult>;
-/**
- * Get aggregate escalation statistics scoped to the user's roles.
- *
- * @param input.period — time window (`1h`, `24h`, `7d`, `30d`)
- * @param auth — authenticated user context
- * @returns `{ status: 200, data: { pending, claimed, created, resolved, by_role, by_type } }`
- */
-export declare function getEscalationStats(input: {
-    period?: string;
-}, auth: LTApiAuth): Promise<LTApiResult>;
-/**
- * Get a single escalation by ID.
- *
- * Non-superadmin users must hold the escalation's assigned role.
- *
- * @param input.id — escalation UUID
- * @param auth — authenticated user context
- * @returns `{ status: 200, data: <escalation record> }` or 403/404
- */
-export declare function getEscalation(input: {
-    id: string;
-}, auth: LTApiAuth): Promise<LTApiResult>;
-/**
- * List all escalations for a given workflow ID.
- *
- * @param input.workflowId — HotMesh workflow ID
- * @returns `{ status: 200, data: { escalations } }`
- */
-export declare function getEscalationsByWorkflowId(input: {
-    workflowId: string;
-}): Promise<LTApiResult>;
-/**
- * Route a pending escalation to a different role.
- *
- * The user must be authorized to escalate from the current role to the
- * target role (checked via escalation chain configuration).
- *
- * @param input.id — escalation UUID
- * @param input.targetRole — destination role
- * @param auth — authenticated user context
- * @returns `{ status: 200, data: <updated escalation> }` or 403/404/409
- */
-export declare function escalateToRole(input: {
-    id: string;
-    targetRole: string;
-}, auth: LTApiAuth): Promise<LTApiResult>;
-/**
- * Claim a pending escalation for the authenticated user.
- *
- * Sets `assigned_to` and `assigned_until` on the escalation (soft lock).
- * Non-superadmin users must hold the escalation's role. Publishes a
- * `escalation.claimed` event.
- *
- * @param input.id — escalation UUID
- * @param input.durationMinutes — claim duration (default: 30)
- * @param auth — authenticated user context
- * @returns `{ status: 200, data: { escalation, isExtension } }` or 403/404/409
- */
-export declare function claimEscalation(input: {
-    id: string;
-    durationMinutes?: number;
-}, auth: LTApiAuth): Promise<LTApiResult>;
-/**
- * Release a claimed escalation back to the pool.
- *
- * Only the user who holds the claim can release it. Publishes a
- * `escalation.released` event.
- *
- * @param input.id — escalation UUID
- * @param auth — authenticated user context
- * @returns `{ status: 200, data: { escalation } }` or 409
- */
-export declare function releaseEscalation(input: {
-    id: string;
-}, auth: LTApiAuth): Promise<LTApiResult>;
-/**
- * Release all escalation claims past their `assigned_until` deadline.
- *
- * Typically called on a maintenance schedule. Returns the count of
- * released claims.
- *
- * @returns `{ status: 200, data: { released: number } }`
- */
-export declare function releaseExpiredClaims(): Promise<LTApiResult>;
-/**
- * Update priority for one or more escalations.
- *
- * @param input.ids — array of escalation UUIDs
- * @param input.priority — new priority (1=critical, 2=high, 3=medium, 4=low)
- * @param auth — authenticated user context (admin or role-holder required)
- * @returns `{ status: 200, data: { updated: number } }`
- */
-export declare function updatePriority(input: {
-    ids: string[];
-    priority: number;
-}, auth: LTApiAuth): Promise<LTApiResult>;
-/**
- * Claim multiple escalations at once for the authenticated user.
- *
- * @param input.ids — array of escalation UUIDs
- * @param input.durationMinutes — claim duration (default: 30)
- * @param auth — authenticated user context
- * @returns `{ status: 200, data: { claimed, skipped } }`
- */
-export declare function bulkClaim(input: {
-    ids: string[];
-    durationMinutes?: number;
-}, auth: LTApiAuth): Promise<LTApiResult>;
-/**
- * Assign multiple escalations to a specific user.
- *
- * Non-superadmin callers must verify the target user holds each
- * escalation's role. Publishes claim events for assigned items.
- *
- * @param input.ids — array of escalation UUIDs
- * @param input.targetUserId — user to assign to
- * @param input.durationMinutes — assignment duration (default: 30)
- * @param auth — authenticated user context
- * @returns `{ status: 200, data: { assigned, skipped } }`
- */
-export declare function bulkAssign(input: {
-    ids: string[];
-    targetUserId: string;
-    durationMinutes?: number;
-}, auth: LTApiAuth): Promise<LTApiResult>;
-/**
- * Route multiple escalations to a different role.
- *
- * @param input.ids — array of escalation UUIDs
- * @param input.targetRole — destination role
- * @param auth — authenticated user context
- * @returns `{ status: 200, data: { updated: number } }`
- */
-export declare function bulkEscalate(input: {
-    ids: string[];
-    targetRole: string;
-}, auth: LTApiAuth): Promise<LTApiResult>;
-/**
- * Trigger AI triage for multiple escalations.
- *
- * Resolves each escalation and starts a triage workflow that uses MCP
- * tools to analyze and potentially auto-resolve the issue.
- *
- * @param input.ids — array of escalation UUIDs
- * @param input.hint — optional natural-language guidance for the triage AI
- * @param auth — authenticated user context
- * @returns `{ status: 200, data: { triaged, workflows } }`
- */
-export declare function bulkTriage(input: {
-    ids: string[];
-    hint?: string;
-}, auth: LTApiAuth): Promise<LTApiResult>;
-/**
- * Resolve a pending escalation with a human-provided payload.
- *
- * Handles two resolution paths:
- * 1. **Signal-routed** — if the escalation has `signal_routing` metadata,
- *    the resolver payload is sent directly to the paused workflow via
- *    `handle.signal()`. Supports both Durable and YAML engines.
- * 2. **Re-run** — the original workflow is re-started with the resolver
- *    payload injected into `envelope.resolver`. The interceptor detects
- *    the re-run and skips to the resolution branch.
- *
- * Password fields in the resolver payload are replaced with ephemeral
- * tokens (15-minute TTL) so plaintext never enters the signal store.
- *
- * Supports optional escalation strategy execution after resolution.
- *
- * @param input.id — escalation UUID
- * @param input.resolverPayload — human decision data
- * @param auth — authenticated user context
- * @returns `{ status: 200, data: { signaled, escalationId, workflowId } }` (signal path)
- *          or `{ status: 200, data: { workflowId, resumed, escalationId } }` (re-run path)
- */
-export declare function resolveEscalation(input: {
-    id: string;
-    resolverPayload: Record<string, any>;
-}, auth: LTApiAuth): Promise<LTApiResult>;