@hotmeshio/long-tail 0.1.6 → 0.1.7

package/docs/events.md

@@ -133,6 +133,34 @@ await eventRegistry.connect();
 
 On disconnect, the adapter calls `drain()` on the NATS connection, ensuring in-flight publishes complete before closing.
 
+## Callback Adapter
+
+`CallbackEventAdapter` delivers events via registered callbacks. Use it when Long Tail runs as an embedded package and you want event subscriptions without socket.io or NATS.
+
+```typescript
+import { eventRegistry, CallbackEventAdapter } from '@hotmeshio/long-tail';
+
+const adapter = new CallbackEventAdapter();
+eventRegistry.register(adapter);
+await adapter.connect();
+
+// Exact match
+const unsub = adapter.on('escalation.claimed', (event) => {
+  console.log('claimed:', event.escalationId);
+});
+
+// Category wildcard — matches all task.* events
+adapter.on('task.*', (event) => { ... });
+
+// Global wildcard
+adapter.on('*', (event) => { ... });
+
+// Unsubscribe
+unsub();
+```
+
+The adapter is registered automatically by `start()`. If you use `createClient()`, the SDK exposes it as `lt.events.on()` — see the [SDK guide](sdk.md) for details.
+
 ## In-Memory Adapter
 
 `InMemoryEventAdapter` captures events in an array. It exists for testing.

package/docs/sdk.md

@@ -0,0 +1,177 @@
+# SDK
+
+Long Tail exposes its full API as an in-process SDK. Every operation available through the REST API can be called directly as a function — same validation, same RBAC, same event publishing — without HTTP transport overhead.
+
+Use the SDK when Long Tail runs as an embedded package inside your own application (NestJS, Next.js, Express, or any Node.js process). The HTTP server, socket.io, and dashboard are optional — disable them and call the SDK directly.
+
+## Setup
+
+```typescript
+import { start, createClient } from '@hotmeshio/long-tail';
+
+// Start without HTTP server
+await start({
+  database: { connectionString: process.env.DATABASE_URL },
+  server: { enabled: false },
+  workers: [{ taskQueue: 'default', workflow: myWorkflow.handler }],
+});
+
+// Create a client with a default auth context
+const lt = createClient({ auth: { userId: 'my-service' } });
+```
+
+The `auth` context binds to every call that requires it. Override per-call when needed:
+
+```typescript
+// Uses the default auth ('my-service')
+await lt.escalations.list({ status: 'pending' });
+
+// Override for a specific call
+await lt.escalations.claim({ id: 'esc_123' }, { userId: 'user-456' });
+```
+
+## LTApiResult
+
+Every SDK call returns an `LTApiResult`:
+
+```typescript
+interface LTApiResult<T = any> {
+  status: number;    // HTTP-equivalent status code (200, 400, 403, 404, 409, 500)
+  data?: T;          // Success payload (present when status is 2xx)
+  error?: string;    // Error message (present when status is 4xx/5xx)
+}
+```
+
+Status codes follow the same semantics as the REST API. Check `result.status` to distinguish success from failure:
+
+```typescript
+const result = await lt.tasks.get({ id: 'task-123' });
+
+if (result.status === 200) {
+  console.log(result.data);  // the task record
+} else if (result.status === 404) {
+  console.log('not found');
+} else {
+  console.error(result.error);
+}
+```
+
+## API Namespaces
+
+The client mirrors the REST route structure:
+
+| Namespace | REST equivalent | Key operations |
+|-----------|----------------|----------------|
+| `lt.tasks` | `/api/tasks` | `list`, `get`, `listProcesses`, `getProcess`, `getProcessStats` |
+| `lt.escalations` | `/api/escalations` | `list`, `listAvailable`, `get`, `claim`, `release`, `resolve`, `escalate`, `bulkClaim`, `bulkAssign` |
+| `lt.workflows` | `/api/workflows` | `invoke`, `getStatus`, `getResult`, `terminate`, `listWorkers`, `listDiscovered`, `listConfigs`, `getConfig`, `upsertConfig` |
+| `lt.yamlWorkflows` | `/api/yaml-workflows` | `list`, `get`, `create`, `deploy`, `activate`, `invoke`, `archive` |
+| `lt.users` | `/api/users` | `list`, `get`, `create`, `update`, `delete`, `getRoles`, `addRole`, `removeRole` |
+| `lt.roles` | `/api/roles` | `list`, `listWithDetails`, `create`, `delete`, `getEscalationChains` |
+| `lt.auth` | `/api/auth` | `login` |
+| `lt.mcp` | `/api/mcp` | `listServers`, `createServer`, `getServer`, `connectServer`, `listTools`, `callTool` |
+| `lt.mcpRuns` | `/api/mcp-runs` | `listEntities`, `listJobs`, `getExecution` |
+| `lt.insight` | `/api/insight` | `mcpQuery`, `buildWorkflow`, `refineWorkflow`, `describeWorkflow` |
+| `lt.exports` | `/api/workflow-states` | `listJobs`, `exportState`, `exportExecution`, `getStatus`, `getState` |
+| `lt.controlplane` | `/api/controlplane` | `listApps`, `rollCall`, `throttle`, `getStreamStats` |
+| `lt.botAccounts` | `/api/bot-accounts` | `list`, `get`, `create`, `update`, `delete`, `listKeys`, `createKey` |
+| `lt.workflowSets` | `/api/workflow-sets` | `create`, `list`, `get`, `updatePlan`, `build`, `deploy` |
+| `lt.settings` | `/api/settings` | `get` |
+| `lt.dba` | `/api/dba` | `prune`, `deploy` |
+| `lt.namespaces` | `/api/namespaces` | `list`, `register` |
+| `lt.maintenance` | `/api/config/maintenance` | `getConfig`, `updateConfig` |
+
+## Events
+
+The SDK includes a callback-based event adapter. Subscribe to events directly — no socket.io client, no WebSocket connection.
+
+```typescript
+// Exact event type
+const unsub = lt.events.on('escalation.claimed', (event) => {
+  console.log('claimed:', event.escalationId, 'by:', event.data?.assigned_to);
+});
+
+// Category wildcard — matches all task.* events
+lt.events.on('task.*', (event) => {
+  metrics.increment(`task.${event.type}`);
+});
+
+// Global wildcard — every event
+lt.events.on('*', (event) => {
+  auditLog.append(event);
+});
+
+// Unsubscribe when done
+unsub();
+```
+
+Events fire from the same publish calls that feed socket.io and NATS. The callback adapter runs in-process with zero serialization overhead. All event types documented in [Events](events.md) are available.
+
+### Event types
+
+```
+task.created    task.started    task.completed    task.escalated    task.failed
+escalation.created    escalation.claimed    escalation.released    escalation.resolved
+workflow.started    workflow.completed    workflow.failed
+activity.started    activity.completed    activity.failed
+milestone
+```
+
+## Running Alongside an HTTP Server
+
+The SDK works whether or not the HTTP server is running. If your application already has its own Express/Fastify/Koa server, you can start Long Tail without its built-in server and expose routes however you like:
+
+```typescript
+// Your NestJS controller
+@Controller('workflows')
+export class WorkflowController {
+  private lt = createClient({ auth: { userId: 'nest-service' } });
+
+  @Post(':type/invoke')
+  async invoke(@Param('type') type: string, @Body() body: any) {
+    return this.lt.workflows.invoke({ type, data: body.data });
+  }
+
+  @Get(':id/status')
+  async status(@Param('id') id: string) {
+    return this.lt.workflows.getStatus({ workflowId: id });
+  }
+}
+```
+
+## CallbackEventAdapter
+
+If you need lower-level control over event subscriptions, use the `CallbackEventAdapter` directly:
+
+```typescript
+import { eventRegistry, CallbackEventAdapter } from '@hotmeshio/long-tail';
+
+const adapter = new CallbackEventAdapter();
+eventRegistry.register(adapter);
+await adapter.connect();
+
+// Subscribe
+const unsub = adapter.on('task.completed', (event) => { ... });
+
+// Unsubscribe
+unsub();
+```
+
+The `createClient()` function manages this automatically — you only need the adapter directly if you want event subscriptions without the full SDK client.
+
+## Mixing Transports
+
+The event registry supports multiple adapters simultaneously. Events publish to all registered adapters in parallel:
+
+```typescript
+await start({
+  database: { connectionString: process.env.DATABASE_URL },
+  server: { enabled: false },
+  events: { nats: { url: 'nats://localhost:4222' } },  // NATS for external consumers
+});
+
+const lt = createClient({ auth: { userId: 'system' } });
+
+// In-process callbacks AND NATS — both receive every event
+lt.events.on('task.*', handleTaskLocally);
+```

package/docs/story.md

@@ -0,0 +1,157 @@
+# The Long Tail Story
+
+## Infrastructure That Learns
+
+Most software you deploy stays exactly as capable as the day you shipped it. It doesn't get smarter. It doesn't absorb what your team knows. It runs the same logic on day one thousand that it ran on day one.
+
+Long Tail is different. Every problem solved through it becomes a permanent capability. The first time, an LLM reasons through it. A human reviews the result. The system compiles the solution into a deterministic tool. The next occurrence runs without AI, without human intervention, without cost. The tool inventory grows. The need for reasoning shrinks. The system gets cheaper and more capable simultaneously.
+
+This document walks through what happens when an organization adopts Long Tail, told from the perspective of what accumulates over time.
+
+---
+
+## The Starting Point
+
+You have Postgres and a problem. Maybe it's content review, document processing, customer onboarding, incident response — any operational workflow where human judgment is currently the bottleneck.
+
+You install the package, point it at your database, and start it up. The system creates its schema, registers the built-in MCP servers, and opens the dashboard. Two container types from one codebase: API servers serve the dashboard and REST endpoints; workers execute workflows. Both read from Postgres.
+
+That's the entire infrastructure story. No message queues to configure. No object stores to provision. No separate orchestration service. Postgres is the durable execution engine, the tool registry, the escalation queue, and the audit log.
+
+---
+
+## The First Tool
+
+The system ships with MCP servers for common capabilities: browser automation, file storage, document vision, database queries, HTTP requests. You can add your own — npm packages, remote services, custom TypeScript — and their tools immediately appear in the designer and compiler.
+
+Open the MCP Tool Designer. Describe what you need in natural language:
+
+> *"Screenshot every page in the admin panel and flag any that return errors."*
+
+The system discovers the relevant tools across registered servers, plans the execution, and runs it. The browser server navigates pages. The vision server inspects screenshots. The file server stores results. Each step is a durable activity — if the process crashes mid-run, it resumes from the last completed step, not from the beginning.
+
+The result lands in your dashboard: a structured output with screenshots, error flags, and metadata. It worked. But it cost tokens, took time, and required an LLM to reason through each step.
+
+This is the dynamic path. It's powerful and flexible. It's also the most expensive way to run anything.
+
+---
+
+## The Compilation Moment
+
+This is the pivot — the point where Long Tail stops being "an AI tool" and starts being infrastructure.
+
+Open the compiler. It examines the execution trace from the dynamic run: which tools were called, in what order, with what arguments. It identifies what's dynamic (the URL, the credentials, the list of pages) versus what's fixed (the CSS selectors, the wait times, the error-detection logic). It produces a deterministic DAG — a directed acyclic graph that encodes the exact same workflow without any LLM reasoning.
+
+Deploy it. It's now a tool — tagged, versioned, invocable. It has typed inputs and outputs. It runs in milliseconds instead of seconds. It costs nothing in API tokens. And it's just as reliable as the dynamic version, because it was extracted from a successful execution, not written by hand.
+
+The compilation step is where operational knowledge crosses the boundary from tacit to executable. Before compilation, the knowledge of "how to screenshot admin pages and check for errors" lived in the LLM prompt and the human who wrote it. After compilation, it lives in the database as a versioned, deterministic tool that anyone — or any other workflow — can invoke.
+
+---
+
+## The Accumulation
+
+Next week, someone describes a similar task. The router examines the request, searches the compiled tool registry by tags and capability descriptions, and finds a match. It extracts the parameters from the natural language request and routes directly to the compiled tool.
+
+No LLM. No tokens. Sub-second execution.
+
+The tool that was born from one person's request now serves the whole team. And it can be composed. Another workflow can call it as a step. The DAG that screenshots pages becomes an activity inside a larger workflow that audits, compares, and reports across environments.
+
+This is the accumulation effect. Each compiled tool is both an endpoint and a building block. The library of tools grows monotonically — new compilations only add capabilities, never remove them. Workflows that compose multiple tools inherit the reliability and speed of each component.
+
+Over time, the system develops a vocabulary of operations specific to your organization. Not generic "send HTTP request" primitives, but domain-specific tools like "audit staging environment" or "validate customer document against policy template." These are your operations, encoded in your database, reflecting your team's actual practices.
+
+---
+
+## The Long Tail
+
+The name comes from the distribution of problems.
+
+Most operational tasks converge to patterns. After a month of use, the common paths — the head of the distribution — are compiled. Content reviews follow templates. Document processing has known formats. Onboarding flows have standard steps. These run deterministically, cheaply, instantly.
+
+What remains is the long tail: novel requests, edge cases, genuine unknowns. A document format nobody's seen before. An admin page with non-standard markup. A customer request that doesn't match any existing workflow.
+
+For these, the dynamic path still runs. The LLM reasons through the problem with the full tool inventory available. When it succeeds, the solution is a candidate for compilation. The tail gets shorter.
+
+When it fails — when the available tools can't solve the problem or the LLM's reasoning produces an incorrect result — the system escalates. RBAC-scoped chains route the problem to the right human based on the credentials, domain, and tool requirements involved. The human resolves it through the dashboard, with full context: what was attempted, what failed, and why.
+
+If the resolution reveals a repeatable pattern, it compiles too. The escalation itself becomes the training data for the next compiled tool.
+
+This is the dynamic the name describes: the long tail of problems that resist automation. Long Tail doesn't eliminate the tail. It systematically shortens it by converting each solved problem into a permanent, reusable capability.
+
+---
+
+## What You Own
+
+Long Tail runs on your Postgres. This isn't a philosophical stance about data sovereignty — it's an architectural consequence of using durable execution backed by a database you control.
+
+The compiled tools live in your database. The execution history lives in your database. The escalation patterns, the credential flows, the audit trail — all in your database. You can back it up, replicate it, query it with SQL, migrate it to another provider, or inspect it with any tool that speaks Postgres.
+
+When the system compiles a workflow, that compiled DAG is a row in a table you own. When a team member resolves an escalation, that resolution is recorded in a schema you can read. The institutional knowledge that accumulates over months of use — the specific ways your organization handles its specific problems — never leaves your infrastructure.
+
+This matters for a practical reason beyond principle. Operational knowledge is one of the few genuine competitive advantages an organization has. The specific patterns of how your team reviews content, processes documents, responds to incidents — these reflect hard-won expertise. Infrastructure that captures and encodes that expertise should store it where you control access, retention, and usage.
+
+Two container types scale independently behind a load balancer. Add API servers for dashboard traffic. Add workers for execution throughput. Both connect to the same Postgres. There's no orchestration service to license, no execution platform to subscribe to, no vendor that accumulates your operational patterns as a side effect of providing the service.
+
+---
+
+## The Pluggable Platform
+
+MCP servers are the extension mechanism. Each server registers a set of tools with typed inputs, outputs, and descriptive metadata. The system uses this metadata for discovery — both by the LLM during dynamic execution and by the router during compiled tool matching.
+
+The built-in servers cover common ground: browser automation for web interaction, file storage for document handling, vision for image analysis, database queries for structured data, HTTP for external API integration. But the real power is in what you add.
+
+Install an npm package that exposes an MCP server. Register it. Its tools are immediately available to the designer, the compiler, and the router. A server that wraps your internal API becomes a set of tools that workflows can compose. A server that connects to a third-party service extends the platform's reach without modifying any core code.
+
+The LLM can even discover what's missing. If a dynamic execution fails because no available tool can perform a required step, the triage system surfaces the gap: "This workflow needs access to [capability] but no registered server provides it." Install the server, register the tools, and re-run. The platform grows in response to the problems it encounters.
+
+Tags organize the tool space. Servers and tools carry tags that describe their domain and capabilities. The router uses tags to scope discovery — a database analytics query doesn't need to search browser automation tools. As the tool inventory grows, tags keep discovery focused and fast.
+
+---
+
+## The Three Execution Tiers
+
+Every request flows through a routing decision:
+
+**Deterministic.** A compiled tool matches the request. Parameters are extracted and the DAG executes directly. No LLM involved. Sub-second. Free.
+
+**Dynamic.** No compiled tool matches, but the available tools can likely solve the problem. The LLM reasons through the execution, calling tools as needed. Durable — survives crashes and restarts. Successful runs are candidates for compilation.
+
+**Escalation.** The dynamic path fails or the system determines it can't solve the problem autonomously. RBAC-scoped routing sends it to the right human with full context. The human resolves it through the dashboard. Resolutions that reveal patterns feed back into compilation.
+
+The ratio shifts over time. Early on, most requests take the dynamic path. As compilations accumulate, more requests route deterministically. The escalation path handles a shrinking residual. The system is always getting cheaper and faster, with no manual optimization required.
+
+---
+
+## The Workflow Lifecycle
+
+A workflow in Long Tail has a natural lifecycle:
+
+**Born dynamic.** Someone describes a need. The LLM orchestrates tools to meet it. The execution is durable — each step is checkpointed in Postgres. If the worker crashes, it resumes from the last completed activity, not from scratch.
+
+**Reviewed.** The result appears in the dashboard. A human confirms it's correct, flags issues, or provides corrections. This review step is optional but valuable — it's the quality gate that ensures compiled tools encode correct behavior.
+
+**Compiled.** The successful execution trace is fed to the compiler. The compiler extracts the fixed structure, parameterizes the dynamic inputs, and produces a deterministic DAG. The DAG is deployed as a new tool with typed inputs, outputs, and tags.
+
+**Composed.** The compiled tool becomes available as an activity for other workflows. A tool that validates a single document can be composed into a workflow that processes a batch. A tool that checks a single page can be composed into a site-wide audit. Composition is how individual capabilities combine into organizational processes.
+
+**Evolved.** When the domain changes — new document formats, new page layouts, new policies — the compiled tool may start failing. Failures route to escalation. The human resolution captures the new pattern. A new compilation encodes the updated behavior. The old version remains available for rollback.
+
+This lifecycle runs continuously. The system is always compiling, always accumulating, always refining. No migration projects. No "automation initiatives." Just a steady conversion of operational knowledge into executable tools.
+
+---
+
+## What You're Really Building
+
+Long Tail isn't a workflow engine that happens to use AI. It's a **knowledge compiler** that uses workflows as the intermediate representation.
+
+The input is human expertise and LLM reasoning — the tacit operational knowledge of how your team solves problems. The output is deterministic, versioned, composable tools that run without either.
+
+Every organization has operational patterns that live in people's heads, in runbooks, in tribal knowledge, in the "ask Sarah, she knows how to handle that" moments. These patterns are valuable, fragile, and almost never systematically captured. When someone leaves, the knowledge leaves with them. When a team scales, the patterns dilute.
+
+Long Tail is the machine that extracts those patterns and makes them executable. Not by asking people to document their processes — that rarely works and the documentation immediately drifts from reality. Instead, by observing how problems are actually solved, compiling the successful patterns, and deploying them as tools that anyone can invoke.
+
+The LLM is scaffolding. It provides the initial reasoning capability that bridges the gap between "describe what you need" and "here's a working solution." But scaffolding is temporary by design. As compiled tools accumulate, the LLM handles less. The deterministic tools handle more. The system converges toward a state where the LLM is only needed for genuinely novel problems — the shrinking long tail.
+
+The compiled tools are the building--the durable artifact. They're what remains after the LLM has done its work and moved on. They encode your organization's operational knowledge in a form that's versioned, testable, composable, and entirely under your control.
+
+That's the Long Tail.