@hotmeshio/long-tail 0.1.5 → 0.1.7

package/docs/story.md

@@ -0,0 +1,157 @@
+# The Long Tail Story
+
+## Infrastructure That Learns
+
+Most software you deploy stays exactly as capable as the day you shipped it. It doesn't get smarter. It doesn't absorb what your team knows. It runs the same logic on day one thousand that it ran on day one.
+
+Long Tail is different. Every problem solved through it becomes a permanent capability. The first time, an LLM reasons through it. A human reviews the result. The system compiles the solution into a deterministic tool. The next occurrence runs without AI, without human intervention, without cost. The tool inventory grows. The need for reasoning shrinks. The system gets cheaper and more capable simultaneously.
+
+This document walks through what happens when an organization adopts Long Tail, told from the perspective of what accumulates over time.
+
+---
+
+## The Starting Point
+
+You have Postgres and a problem. Maybe it's content review, document processing, customer onboarding, incident response — any operational workflow where human judgment is currently the bottleneck.
+
+You install the package, point it at your database, and start it up. The system creates its schema, registers the built-in MCP servers, and opens the dashboard. Two container types from one codebase: API servers serve the dashboard and REST endpoints; workers execute workflows. Both read from Postgres.
+
+That's the entire infrastructure story. No message queues to configure. No object stores to provision. No separate orchestration service. Postgres is the durable execution engine, the tool registry, the escalation queue, and the audit log.
+
+---
+
+## The First Tool
+
+The system ships with MCP servers for common capabilities: browser automation, file storage, document vision, database queries, HTTP requests. You can add your own — npm packages, remote services, custom TypeScript — and their tools immediately appear in the designer and compiler.
+
+Open the MCP Tool Designer. Describe what you need in natural language:
+
+> *"Screenshot every page in the admin panel and flag any that return errors."*
+
+The system discovers the relevant tools across registered servers, plans the execution, and runs it. The browser server navigates pages. The vision server inspects screenshots. The file server stores results. Each step is a durable activity — if the process crashes mid-run, it resumes from the last completed step, not from the beginning.
+
+The result lands in your dashboard: a structured output with screenshots, error flags, and metadata. It worked. But it cost tokens, took time, and required an LLM to reason through each step.
+
+This is the dynamic path. It's powerful and flexible. It's also the most expensive way to run anything.
+
+---
+
+## The Compilation Moment
+
+This is the pivot — the point where Long Tail stops being "an AI tool" and starts being infrastructure.
+
+Open the compiler. It examines the execution trace from the dynamic run: which tools were called, in what order, with what arguments. It identifies what's dynamic (the URL, the credentials, the list of pages) versus what's fixed (the CSS selectors, the wait times, the error-detection logic). It produces a deterministic DAG — a directed acyclic graph that encodes the exact same workflow without any LLM reasoning.
+
+Deploy it. It's now a tool — tagged, versioned, invocable. It has typed inputs and outputs. It runs in milliseconds instead of seconds. It costs nothing in API tokens. And it's just as reliable as the dynamic version, because it was extracted from a successful execution, not written by hand.
+
+The compilation step is where operational knowledge crosses the boundary from tacit to executable. Before compilation, the knowledge of "how to screenshot admin pages and check for errors" lived in the LLM prompt and the human who wrote it. After compilation, it lives in the database as a versioned, deterministic tool that anyone — or any other workflow — can invoke.
+
+---
+
+## The Accumulation
+
+Next week, someone describes a similar task. The router examines the request, searches the compiled tool registry by tags and capability descriptions, and finds a match. It extracts the parameters from the natural language request and routes directly to the compiled tool.
+
+No LLM. No tokens. Sub-second execution.
+
+The tool that was born from one person's request now serves the whole team. And it can be composed. Another workflow can call it as a step. The DAG that screenshots pages becomes an activity inside a larger workflow that audits, compares, and reports across environments.
+
+This is the accumulation effect. Each compiled tool is both an endpoint and a building block. The library of tools grows monotonically — new compilations only add capabilities, never remove them. Workflows that compose multiple tools inherit the reliability and speed of each component.
+
+Over time, the system develops a vocabulary of operations specific to your organization. Not generic "send HTTP request" primitives, but domain-specific tools like "audit staging environment" or "validate customer document against policy template." These are your operations, encoded in your database, reflecting your team's actual practices.
+
+---
+
+## The Long Tail
+
+The name comes from the distribution of problems.
+
+Most operational tasks converge to patterns. After a month of use, the common paths — the head of the distribution — are compiled. Content reviews follow templates. Document processing has known formats. Onboarding flows have standard steps. These run deterministically, cheaply, instantly.
+
+What remains is the long tail: novel requests, edge cases, genuine unknowns. A document format nobody's seen before. An admin page with non-standard markup. A customer request that doesn't match any existing workflow.
+
+For these, the dynamic path still runs. The LLM reasons through the problem with the full tool inventory available. When it succeeds, the solution is a candidate for compilation. The tail gets shorter.
+
+When it fails — when the available tools can't solve the problem or the LLM's reasoning produces an incorrect result — the system escalates. RBAC-scoped chains route the problem to the right human based on the credentials, domain, and tool requirements involved. The human resolves it through the dashboard, with full context: what was attempted, what failed, and why.
+
+If the resolution reveals a repeatable pattern, it compiles too. The escalation itself becomes the training data for the next compiled tool.
+
+This is the dynamic the name describes: the long tail of problems that resist automation. Long Tail doesn't eliminate the tail. It systematically shortens it by converting each solved problem into a permanent, reusable capability.
+
+---
+
+## What You Own
+
+Long Tail runs on your Postgres. This isn't a philosophical stance about data sovereignty — it's an architectural consequence of using durable execution backed by a database you control.
+
+The compiled tools live in your database. The execution history lives in your database. The escalation patterns, the credential flows, the audit trail — all in your database. You can back it up, replicate it, query it with SQL, migrate it to another provider, or inspect it with any tool that speaks Postgres.
+
+When the system compiles a workflow, that compiled DAG is a row in a table you own. When a team member resolves an escalation, that resolution is recorded in a schema you can read. The institutional knowledge that accumulates over months of use — the specific ways your organization handles its specific problems — never leaves your infrastructure.
+
+This matters for a practical reason beyond principle. Operational knowledge is one of the few genuine competitive advantages an organization has. The specific patterns of how your team reviews content, processes documents, responds to incidents — these reflect hard-won expertise. Infrastructure that captures and encodes that expertise should store it where you control access, retention, and usage.
+
+Two container types scale independently behind a load balancer. Add API servers for dashboard traffic. Add workers for execution throughput. Both connect to the same Postgres. There's no orchestration service to license, no execution platform to subscribe to, no vendor that accumulates your operational patterns as a side effect of providing the service.
+
+---
+
+## The Pluggable Platform
+
+MCP servers are the extension mechanism. Each server registers a set of tools with typed inputs, outputs, and descriptive metadata. The system uses this metadata for discovery — both by the LLM during dynamic execution and by the router during compiled tool matching.
+
+The built-in servers cover common ground: browser automation for web interaction, file storage for document handling, vision for image analysis, database queries for structured data, HTTP for external API integration. But the real power is in what you add.
+
+Install an npm package that exposes an MCP server. Register it. Its tools are immediately available to the designer, the compiler, and the router. A server that wraps your internal API becomes a set of tools that workflows can compose. A server that connects to a third-party service extends the platform's reach without modifying any core code.
+
+The LLM can even discover what's missing. If a dynamic execution fails because no available tool can perform a required step, the triage system surfaces the gap: "This workflow needs access to [capability] but no registered server provides it." Install the server, register the tools, and re-run. The platform grows in response to the problems it encounters.
+
+Tags organize the tool space. Servers and tools carry tags that describe their domain and capabilities. The router uses tags to scope discovery — a database analytics query doesn't need to search browser automation tools. As the tool inventory grows, tags keep discovery focused and fast.
+
+---
+
+## The Three Execution Tiers
+
+Every request flows through a routing decision:
+
+**Deterministic.** A compiled tool matches the request. Parameters are extracted and the DAG executes directly. No LLM involved. Sub-second. Free.
+
+**Dynamic.** No compiled tool matches, but the available tools can likely solve the problem. The LLM reasons through the execution, calling tools as needed. Durable — survives crashes and restarts. Successful runs are candidates for compilation.
+
+**Escalation.** The dynamic path fails or the system determines it can't solve the problem autonomously. RBAC-scoped routing sends it to the right human with full context. The human resolves it through the dashboard. Resolutions that reveal patterns feed back into compilation.
+
+The ratio shifts over time. Early on, most requests take the dynamic path. As compilations accumulate, more requests route deterministically. The escalation path handles a shrinking residual. The system is always getting cheaper and faster, with no manual optimization required.
+
+---
+
+## The Workflow Lifecycle
+
+A workflow in Long Tail has a natural lifecycle:
+
+**Born dynamic.** Someone describes a need. The LLM orchestrates tools to meet it. The execution is durable — each step is checkpointed in Postgres. If the worker crashes, it resumes from the last completed activity, not from scratch.
+
+**Reviewed.** The result appears in the dashboard. A human confirms it's correct, flags issues, or provides corrections. This review step is optional but valuable — it's the quality gate that ensures compiled tools encode correct behavior.
+
+**Compiled.** The successful execution trace is fed to the compiler. The compiler extracts the fixed structure, parameterizes the dynamic inputs, and produces a deterministic DAG. The DAG is deployed as a new tool with typed inputs, outputs, and tags.
+
+**Composed.** The compiled tool becomes available as an activity for other workflows. A tool that validates a single document can be composed into a workflow that processes a batch. A tool that checks a single page can be composed into a site-wide audit. Composition is how individual capabilities combine into organizational processes.
+
+**Evolved.** When the domain changes — new document formats, new page layouts, new policies — the compiled tool may start failing. Failures route to escalation. The human resolution captures the new pattern. A new compilation encodes the updated behavior. The old version remains available for rollback.
+
+This lifecycle runs continuously. The system is always compiling, always accumulating, always refining. No migration projects. No "automation initiatives." Just a steady conversion of operational knowledge into executable tools.
+
+---
+
+## What You're Really Building
+
+Long Tail isn't a workflow engine that happens to use AI. It's a **knowledge compiler** that uses workflows as the intermediate representation.
+
+The input is human expertise and LLM reasoning — the tacit operational knowledge of how your team solves problems. The output is deterministic, versioned, composable tools that run without either.
+
+Every organization has operational patterns that live in people's heads, in runbooks, in tribal knowledge, in the "ask Sarah, she knows how to handle that" moments. These patterns are valuable, fragile, and almost never systematically captured. When someone leaves, the knowledge leaves with them. When a team scales, the patterns dilute.
+
+Long Tail is the machine that extracts those patterns and makes them executable. Not by asking people to document their processes — that rarely works and the documentation immediately drifts from reality. Instead, by observing how problems are actually solved, compiling the successful patterns, and deploying them as tools that anyone can invoke.
+
+The LLM is scaffolding. It provides the initial reasoning capability that bridges the gap between "describe what you need" and "here's a working solution." But scaffolding is temporary by design. As compiled tools accumulate, the LLM handles less. The deterministic tools handle more. The system converges toward a state where the LLM is only needed for genuinely novel problems — the shrinking long tail.
+
+The compiled tools are the building--the durable artifact. They're what remains after the LLM has done its work and moved on. They encode your organization's operational knowledge in a form that's versioned, testable, composable, and entirely under your control.
+
+That's the Long Tail.

package/docs/workflow-builder.md

@@ -0,0 +1,371 @@
+# The Workflow Builder: From Prompt to Plan
+
+## The Gap Between a Prompt and a System
+
+The current workflow builder takes a prompt and produces a single DAG. Describe a task, the LLM constructs YAML, you deploy a tool. This works when the unit of work is one workflow — screenshot a page, check a patient's coverage, submit a form.
+
+But the engineering team sitting with Linda doesn't have one task. They have a PRD. A technical design document. Pages of specifications describing a referral intake system with branching paths, composed sub-processes, human review gates, and payer-specific variations. The system they need isn't one workflow. It's a set of interrelated workflows — some calling others, some waiting for human input, some branching based on data conditions — that together implement a complete operational process.
+
+Today, the engineer would decompose the PRD manually, build each workflow individually through the wizard, deploy them into the same namespace, and wire the composition by hand. That works, but it's the same translation problem the builder was designed to eliminate: the engineer becomes the bottleneck between the specification and the executable system.
+
+The evolution is plan mode. When the input exceeds what a single workflow can express — because it's long, because it describes composition, because it references multiple processes — the builder enters a planning phase. It decomposes the specification into workflows, identifies their relationships, builds them leaf-first, and deploys them as a coordinated set.
+
+---
+
+## Two Entry Points, Same Destination
+
+The builder already has two modes:
+
+**Discover & Compile.** Run a dynamic execution with AI, then compile the trace into a deterministic DAG. The execution is the specification — the compiler extracts the pattern from what actually happened.
+
+**Direct Build.** Describe the workflow, and the LLM constructs the YAML directly from tool schemas without executing anything. The engineer's description is the specification.
+
+Plan mode extends Direct Build. The input is the same textarea. The difference is what the input contains. A short prompt ("check referral coverage for a patient") produces a single workflow. A long specification with composition, branching, and multiple processes triggers plan mode.
+
+The detection is straightforward: length beyond a threshold, or structural signals in the content — references to multiple steps that are themselves processes, explicit composition language ("this flow calls that flow"), mentions of planning or phased execution. When plan mode activates, the builder wraps the existing single-workflow wizard in an outer loop that plans first, then builds each workflow through the same pipeline.
+
+---
+
+## What Plan Mode Produces
+
+A plan is a decomposition of the specification into workflows with defined relationships:
+
+**Leaf workflows** have no dependencies on other custom workflows. They call MCP tools directly — FHIR queries, browser actions, file operations. They're the atomic units of the system.
+
+**Composition workflows** call leaf workflows (or other composition workflows) as activities. They encode sequencing, branching, and human-in-the-loop gates. They're the orchestration layer.
+
+**The build order is leaf-first.** The planner identifies terminal workflows — the ones that don't invoke other custom workflows — and builds those first. Then it builds the workflows that compose them. Then the workflows that compose those. The tree builds from the leaves up, because each layer needs to reference the tools that already exist in the layer below.
+
+Each workflow in the plan belongs to a namespace. Workflows in the same namespace are deployed together as one HotMesh application. They can invoke each other using HotMesh's `await` activity — a direct, typed call within the same execution engine. This is the tightest composition: same deployment, same transaction boundary, same namespace.
+
+Workflows in different namespaces invoke each other as MCP tools. The calling workflow treats the target as any other tool — discovered by tag, invoked through the MCP protocol, results returned as structured data. This is the loosest composition: independent deployment, independent versioning, connected only through the tool interface.
+
+The planner decides which workflows belong together based on coupling. Workflows that share data context, that are always deployed together, that represent phases of the same process — same namespace. Workflows that represent independent capabilities, that might be reused across processes, that have different versioning cycles — separate namespaces.
+
+---
+
+## The Outer Loop
+
+The existing wizard has four steps in builder mode: Describe, Profile, Deploy, Test. Plan mode wraps this in an outer loop:
+
+**Plan step.** The specification goes in. The planner produces a structured decomposition: which workflows to build, in what order, with what relationships. The dashboard displays this as a list or graph — the set of workflows, their dependencies, their namespaces. The engineer reviews, adjusts, and confirms.
+
+**Build loop.** For each workflow in the plan (leaf-first), the system runs the existing builder pipeline: construct YAML from the specification slice, validate, present for review. The engineer sees the familiar Profile → Deploy → Test sequence for each workflow, but the context is richer — the plan shows where this workflow fits in the larger system, which workflows call it, which it calls.
+
+**Verification.** After all workflows are built and deployed, the system runs the composition end-to-end. The engineer provides inputs at the top level, and the execution cascades through the composed workflows. Escalation points pause and wait. Branch conditions route correctly. The full system operates as specified.
+
+The UX for the outer loop is familiar. The plan step is a new panel above the wizard — a progress view showing which workflows have been built, which are next, which are pending. When the engineer clicks into a specific workflow, they see the standard wizard steps. When they step back out, they see the plan.
+
+Real-time updates flow through the existing event system. As each workflow compiles, deploys, and activates, the plan view updates. Build progress, deployment status, test results — all surfaced through the same SSE/NATS topic subscriptions the wizard already uses.
+
+Plan view has both expanded, collapsed modes. These serve to infrorm what is going on at a global level, but also can be collapsed sufficiently to provide high-level progress while allowing the engineer to focus on the target workflow currently loaded in the wizard.
+
+---
+
+## Workflow Sets: The Data Model
+
+The current `lt_yaml_workflows` table stores individual workflows. Plan mode produces sets of related workflows. The relationship between them — which calls which, which namespace they share, which plan produced them — needs a home.
+
+A new entity: **workflow set**. A set is a named group of workflows produced by a single plan. It records:
+
+- The original specification (the PRD, the TDD, the markdown that was pasted in)
+- The plan (the decomposition into workflows with relationships)
+- The namespaces involved
+- The build order
+- The status of each workflow in the set (planned, building, deployed, active, failed)
+
+Each `lt_yaml_workflows` record gains an optional `set_id` foreign key. Workflows not produced by a plan have no set. Workflows produced by a plan reference their set, and through it, their siblings and their relationships.
+
+The set is a first-class entity in the dashboard. An engineer can view all workflows in a set, see their dependency graph, deploy or redeploy the set, and trace execution through composed workflows. When a leaf workflow is updated (because Epic changed an API, because a payer changed rules), the set view shows which composition workflows are affected.
+
+This also solves the grouping problem for workflows that aren't plan-produced. An engineer who builds three related workflows individually can group them into a set after the fact. The set is an organizational unit, not just a plan artifact.
+
+---
+
+## Composition Mechanics
+
+### Same-Namespace Composition
+
+Workflows in the same namespace are deployed as a single HotMesh application. They share a deployment lifecycle — when any workflow in the namespace is updated, the entire namespace redeploys (merging all YAML graphs).
+
+Within the same namespace, a workflow invokes another using a worker activity whose `workflowName` matches the target's `subscribes` topic. The call is synchronous from the caller's perspective — the activity blocks until the child workflow completes — but durable underneath. If the worker process crashes, the child workflow's progress is preserved, and execution resumes from the last checkpoint.
+
+Data flows through typed input/output schemas. The caller maps its data to the child's input schema. The child's output maps back to the caller's next activity. The YAML wiring is explicit — every field is traced.
+
+### Cross-Namespace Composition
+
+Workflows in different namespaces are independent MCP tools. The calling workflow invokes them through the standard MCP tool protocol — the same mechanism used for browser automation, FHIR queries, or any other tool.
+
+This is composition through the tool layer. The caller doesn't know or care whether the target is a compiled workflow, a built-in MCP server, or an external service. It discovers the tool by tag, passes arguments, and receives results. The target workflow runs in its own namespace, with its own deployment lifecycle, its own versioning.
+
+Cross-namespace composition is the right choice when:
+
+- The target workflow is reusable across processes (a coverage check used by both intake and scheduling)
+- The target has a different release cadence (payer-specific tools updated independently)
+- The target belongs to a different team or domain
+
+### The Composition Spectrum
+
+Same-namespace `await` and cross-namespace MCP tool calls are two points on a spectrum. The planner chooses based on coupling signals in the specification. Tightly coupled processes that share data context and deploy together go in the same namespace. Independent capabilities that serve multiple consumers go in separate namespaces.
+
+An engineer can override the planner's choice. Move a workflow from one namespace to another, and the composition mechanism changes automatically — `await` becomes an MCP tool call, or vice versa. The workflow's logic doesn't change. Only the wiring.
+
+---
+
+## The Prompt Strategy
+
+The builder's LLM prompt is the critical path. It teaches the model how to construct valid HotMesh YAML — activity definitions, transitions, `@pipe` data flow, signal hooks, iteration patterns, collision-proofing suffixes. The existing prompt handles single workflows well.
+
+Plan mode extends the prompt in two dimensions:
+
+**Decomposition.** The planner prompt takes the full specification and produces a structured plan: workflow names, descriptions, input/output contracts, dependencies, namespace assignments, build order. This is a different LLM task than YAML construction — it's architectural reasoning about how to decompose a system into workflows.
+
+**Cross-references.** When the builder constructs a composition workflow, it needs to know the input/output schemas of the workflows it calls. The prompt includes these schemas as context — "workflow `check-coverage` accepts `{patient_id, referral_id}` and returns `{covered, prior_auth_required, plan_code}`." The builder wires the composition using these contracts.
+
+The prompt strategy for complex specifications is iterative. The planner makes a first pass. The engineer reviews. Adjustments feed back into the planner. Each workflow in the plan is built with the full plan as context — the builder knows where each piece fits. This is why plan mode is a loop, not a single shot.
+
+For very large specifications — a full PRD with edge cases, error handling, and operational requirements — the planner may produce nested plans. A top-level plan decomposes into sub-systems. Each sub-system has its own namespace and its own set of workflows. The sub-systems compose through MCP tool calls. This is a plan that produces plans — the outer loop runs recursively.
+
+---
+
+## Human-in-the-Loop in Compiled Workflows
+
+The existing system ships a human-queue MCP server with tools for durable pause points: `escalate_to_human`, `check_resolution`, `escalate_and_wait`. Compiled YAML workflows use these as regular activities.
+
+In the YAML, a human-in-the-loop step is two activities:
+
+1. A **worker** activity that calls `escalate_to_human` (or `escalate_and_wait`) with the escalation payload — what needs review, what context to show, what role should handle it.
+2. A **hook** activity that pauses the workflow and waits for a signal. The signal carries the human's response — approved/rejected, corrected data, additional instructions.
+
+The hook's `signal_schema` defines what the human provides. The dashboard renders a form from this schema. When the human submits, the signal fires, the hook receives the data, and the workflow resumes.
+
+Plan mode makes this explicit in the decomposition. When the specification says "a coordinator reviews the referral before scheduling," the planner creates a workflow with an escalation step between the validation activities and the scheduling activities. The escalation's role, payload, and expected response are derived from the specification.
+
+Conditional transitions after the hook encode the branching: if approved, proceed to scheduling; if rejected, escalate to the referring provider; if corrected, re-run validation with the corrected data. These branches are deterministic — the YAML encodes every path. The human's choice selects the path. No LLM involved.
+
+---
+
+## What the Engineer Sees
+
+### Pasting the PRD
+
+The engineer opens the Pipeline Designer in builder mode. The textarea is the same one that accepts a short prompt. This time, the engineer pastes three pages of markdown — a referral intake specification with sections for validation, coverage checking, document requirements, payer-specific rules, escalation chains, and scheduling handoff.
+
+The system detects the input's complexity and activates plan mode. The UI transitions: above the wizard steps, a new panel appears showing the plan as it forms.
+
+### The Plan View
+
+The planner produces a decomposition. The dashboard renders it as a dependency graph:
+
+```
+referral-intake (namespace: referral-ops)
+├── validate-referral (leaf)
+├── check-coverage (leaf, payer-branching)
+├── verify-documents (leaf, referral-type-specific)
+├── resolve-gaps (composition: escalate → wait → retry)
+├── route-to-scheduling (leaf)
+└── process-referral (composition: orchestrates all above)
+```
+
+Each node shows: name, type (leaf/composition), namespace, status (planned/building/deployed). The engineer can click into any node to see the planned input/output schema and which tools it will use.
+
+The engineer reviews. "The coverage check should be in its own namespace — we reuse it for eligibility verification in another process." They drag the node to a new namespace. The plan adjusts: the composition workflow that called it now references it as an MCP tool instead of an `await` activity.
+
+### Building Leaf-First
+
+The engineer confirms the plan. The system begins building, starting with leaf workflows. The plan view updates in real time — each node's status changes from "planned" to "building" to "deployed" as the builder completes each workflow.
+
+For each workflow, the engineer can expand into the standard wizard: review the YAML, adjust the profile, deploy, test with sample inputs. Or they can let the builder proceed automatically and review the full set at the end.
+
+### Testing the Composition
+
+After all workflows deploy, the engineer runs the top-level `process-referral` workflow with a test referral. The execution cascades: validation calls the validate-referral tool, coverage check invokes the check-coverage MCP tool (cross-namespace), document verification runs in-namespace. At the escalation step, the workflow pauses. The engineer resolves it through the dashboard. The workflow resumes and completes.
+
+The plan view shows the execution trace across all workflows — a unified timeline of which tools ran in which workflow, with durations and data flow visible.
+
+---
+
+## The Epic Story, Continued
+
+The engineering team building referral intake for back-office customers doesn't paste one prompt at a time. They have specifications. They've sat with Linda and Maria and James, captured the full intake process, and written it up as a PRD with sections, edge cases, and decision tables.
+
+They paste the PRD into the builder. Plan mode activates. The planner decomposes it:
+
+- **Namespace `referral-intake`**: `validate-referral`, `verify-documents`, `resolve-gaps`, `route-to-scheduling`, `process-referral` (composition)
+- **Namespace `payer-tools`**: `check-coverage-bluecross`, `check-coverage-aetna`, `check-coverage-default`, `check-coverage` (router composition)
+- **Namespace `epic-fhir`**: Low-level FHIR operations if not already registered as MCP tools
+
+The payer-specific coverage tools are in their own namespace because they version independently — when Aetna changes rules, only `check-coverage-aetna` updates. The intake workflows reference them as MCP tools.
+
+The engineer reviews the plan, adjusts namespace assignments, confirms. The builder produces the full set of workflows. The engineer tests the composition end-to-end against Epic's sandbox. Linda's knowledge, Maria's knowledge, James's knowledge — encoded in a coordinated set of deterministic tools that run without AI, without the domain experts, against any customer's Epic instance.
+
+When the PRD changes — and it will — the engineer pastes the updated section. Plan mode produces a delta: which workflows need rebuilding, which are unchanged, which have new dependencies. The system evolves incrementally, the same way the specification evolved.
+
+---
+
+## Implementation Surface
+
+### Data Model
+
+**New table: `lt_workflow_sets`**
+
+| Column | Type | Purpose |
+|--------|------|---------|
+| `id` | UUID | Primary key |
+| `name` | TEXT | Human-readable set name |
+| `description` | TEXT | Set description |
+| `specification` | TEXT | Original input (PRD, TDD, markdown) |
+| `plan` | JSONB | Decomposition: workflows, relationships, namespaces, build order |
+| `namespaces` | TEXT[] | App IDs involved |
+| `status` | TEXT | `planning` / `building` / `deployed` / `active` / `failed` |
+| `source_workflow_id` | TEXT | Builder workflow that produced the plan |
+| `created_at` | TIMESTAMPTZ | |
+| `updated_at` | TIMESTAMPTZ | |
+
+**Addition to `lt_yaml_workflows`:**
+
+| Column | Type | Purpose |
+|--------|------|---------|
+| `set_id` | UUID (nullable, FK) | References `lt_workflow_sets.id` |
+| `set_role` | TEXT (nullable) | `leaf` / `composition` / `router` |
+| `set_build_order` | INTEGER (nullable) | Build sequence within the set |
+
+### API Endpoints
+
+**Plan lifecycle:**
+- `POST /api/workflow-sets` — create set from specification, triggers planning
+- `GET /api/workflow-sets` — list sets with status filters
+- `GET /api/workflow-sets/:id` — get set with plan and workflow statuses
+- `PUT /api/workflow-sets/:id/plan` — update plan (engineer adjustments)
+- `POST /api/workflow-sets/:id/build` — start building (leaf-first)
+- `POST /api/workflow-sets/:id/deploy` — deploy all namespaces
+- `GET /api/workflow-sets/:id/workflows` — list workflows in set with relationships
+
+**Extended workflow endpoints:**
+- `GET /api/yaml-workflows?set_id=:id` — filter by set membership
+
+### Workflow: `mcpWorkflowPlanner`
+
+A new system workflow that decomposes specifications into plans. Activities:
+
+- `analyzeSpecification` — detect complexity, extract structural signals
+- `decomposeIntoWorkflows` — LLM-driven decomposition with namespace assignment
+- `validatePlan` — check for circular dependencies, missing contracts, namespace conflicts
+- `refinePlan` — incorporate engineer feedback
+
+The planner workflow produces the plan. The existing `mcpWorkflowBuilder` workflow builds each individual workflow in the plan. The orchestration between planner and builder is itself a composition — the planner calls the builder for each workflow in the plan, leaf-first.
+
+### Dashboard Components
+
+**PlanView** — the outer loop panel showing the workflow dependency graph, build status, and namespace groupings. Wraps the existing wizard steps.
+
+**PlanNode** — individual workflow in the plan view. Shows name, type, namespace, status, and links to the standard wizard for that workflow.
+
+**SetListPage** — list of workflow sets, filterable by status and namespace. Entry point for managing multi-workflow systems.
+
+### Events
+
+Plan mode publishes events on the existing NATS topic space:
+
+- `lt.plan.created` — plan produced from specification
+- `lt.plan.workflow.building` — individual workflow build started
+- `lt.plan.workflow.deployed` — individual workflow deployed
+- `lt.plan.completed` — all workflows in plan built and deployed
+- `lt.plan.failed` — build failure with context
+
+The dashboard subscribes to these for real-time plan view updates.
+
+---
+
+## Prompt Strategy: Teaching the LLM to Build YAML
+
+### What Exists
+
+The current builder prompt (`prompts.ts`) teaches four activity types — trigger, worker, hook, cycle — with strong coverage of `@pipe` RPN semantics and collision-proofing rules. The RPN section ("operands THEN operator") is particularly effective; LLMs reliably produce correct pipe expressions when the stack-machine framing is clear.
+
+The prompt includes an abbreviated operator list: the most-used methods from `@string`, `@date`, `@math`, `@number`, `@array`, `@object`, `@conditional`, and `@json`. This is sufficient for single workflows that transform strings, build paths, and iterate arrays.
+
+### What's Missing
+
+Three activity types are absent from the prompt: **await**, **signal**, and **interrupt**. These are precisely the types needed for composition and cross-workflow communication — the core of plan mode.
+
+The operator catalog is abbreviated. The full HotMesh pipe system exposes 13 handler categories with over 150 methods. Most are rarely needed, but gaps in the catalog produce workarounds: the LLM constructs multi-step pipes to accomplish what a single operator would do, or worse, hardcodes values that should be computed.
+
+Conditional transitions — the mechanism for branching based on activity output — are not documented in the prompt. The LLM can produce linear flows but struggles with branches, error routing, and cycle exit conditions.
+
+### The Reference System
+
+Rather than inflating the prompt with everything, the builder uses a modular reference system. Reference files live alongside the prompt at `system/workflows/mcp-workflow-builder/reference/`:
+
+**`activity-types.md`** — Complete documentation of all 8 activity types with YAML examples, properties, and usage guidance. Covers trigger, worker, hook (all 4 modes), cycle, await (sync and fire-and-forget), signal (one and all), and interrupt (self and remote).
+
+**`pipe-functions.md`** — Full catalog of every `@pipe` operator across all 13 handler categories with signatures and return types.
+
+### Injection Strategy
+
+The prompt is assembled from layers. The base prompt always includes:
+
+- Trigger, worker, hook, cycle — the types every workflow needs
+- `@pipe` RPN rules with the "common mistake" examples
+- Core operators: `@string` (concat, substring, toLowerCase), `@date` (now, yyyymmdd, toISOString), `@math` (add), `@array` (get, length), `@object` (get)
+- Construction rules (collision-proofing, _scope threading, workflowName, job.maps)
+- Clarification protocol
+
+Additional context is injected based on what the specification requires:
+
+**Composition detected** (specification references child workflows, sub-processes, or the plan contains multiple workflows) → inject the **await** activity reference. This teaches the LLM how to invoke child workflows within the same namespace, how to wire input/output contracts, and how to use fire-and-forget mode.
+
+**HITL detected** (specification mentions human review, approval gates, escalation) → inject the **signal** activity reference and the full **hook** web-hook mode documentation. This teaches signal routing through the `hooks:` section, `code: 200` vs `code: 202`, and the escalation-then-wait pattern.
+
+**Cancellation detected** (specification mentions timeout, abort, cancel) → inject the **interrupt** activity reference. Self-interrupt for validation failures, remote interrupt for cancelling child workflows.
+
+**Branching detected** (specification mentions conditional paths, error handling, routing) → inject **conditional transitions** documentation with examples of `conditions.code`, `conditions.match`, and multi-target transition arrays.
+
+**Complex transforms detected** (specification involves date arithmetic, array manipulation, object construction beyond simple field references) → inject the extended **pipe function catalog** for the relevant categories.
+
+### Why Not Include Everything
+
+Prompt length directly affects YAML quality. The current prompt is approximately 270 lines — tight enough that the LLM reads and applies every rule. Adding the full activity type reference (~250 lines) and the full pipe catalog (~300 lines) would triple the prompt size. The LLM would skim, miss construction rules, and produce more errors.
+
+The injection strategy keeps the prompt focused on what this specific workflow needs. A simple three-step linear workflow gets the base prompt. A multi-workflow composition with escalation gates gets the base prompt plus await, signal, hook web-hook mode, and conditional transitions. The prompt grows with complexity, not unconditionally.
+
+### Refinement Loop
+
+When a built workflow fails testing, the `REFINE_PROMPT` guides correction. The refinement prompt can also inject activity-type references that the initial build missed — if the failure is an incorrect await configuration, the await reference is added to the refinement context. The system learns which references each workflow type needs through the feedback loop.
+
+### Enrichments to the Existing Prompt
+
+Based on the full HotMesh documentation, these additions improve accuracy without significant length:
+
+1. **Conditional transitions.** Add a section showing multi-target transitions with `conditions`:
+   ```yaml
+   transitions:
+     check_status:
+       - to: handle_error
+         conditions:
+           code: 500
+       - to: proceed
+   ```
+   This is essential for branching workflows and currently undocumented.
+
+2. **Extended pipe operators in the abbreviated list.** Add: `@string.endsWith`, `@string.startsWith`, `@string.padStart`, `@string.replace`, `@array.join`, `@array.sort`, `@array.slice`, `@object.create`, `@object.assign`, `@object.entries`, `@object.fromEntries`, `@conditional.nullish`, `@conditional.equality`, `@conditional.strict_equality`, `@number.isInteger`, `@number.parseFloat`, `@number.parseInt`, `@number.toFixed`. These appear frequently in real workflow specifications and their absence forces workarounds.
+
+3. **Hook sleep mode.** The existing prompt mentions hook as "durable pause point" but doesn't show the sleep configuration. Add: `sleep: 5` for fixed delays, `@pipe` expression for dynamic delays.
+
+4. **`stats` on trigger.** Custom job IDs and key-based lookups are needed for idempotent workflows. Add the `stats.id` and `stats.key` pattern.
+
+5. **`await: false` pattern.** Fire-and-forget child workflows are common in composition. A one-line addition to the hook/cycle section would cover it: "For fire-and-forget child workflows, use `type: await` with `await: false`."
+
+These additions total roughly 30 lines — minimal prompt growth for significant capability gain.
+
+---
+
+## What This Enables
+
+An engineer with a specification — a PRD, a TDD, a markdown document describing a multi-step operational process — pastes it into the builder and gets back a coordinated set of deterministic workflows. Leaf workflows call MCP tools. Composition workflows orchestrate leaves. Human-in-the-loop gates pause for review. Payer-specific branches route correctly. The full system deploys as a set of tools that other workflows can compose further.
+
+The specification is the input. The compiled DAGs are the output. Plan mode is the compiler that bridges the gap. The engineer reviews and adjusts at every step — this isn't autonomous generation. It's assisted decomposition, where the LLM handles the YAML construction and the engineer handles the architectural decisions.
+
+The citizen developer illusion dissolves. What remains is an engineer who understands the domain, understands composition, and has a tool that translates specifications into executable workflows without writing YAML by hand. The tool handles the syntax. The engineer handles the semantics.
+
+And because every compiled workflow is an MCP tool, the output of one plan can be the input of another. A set of referral intake workflows becomes a tool that a scheduling system calls. A set of payer verification workflows becomes a tool that an eligibility checker calls. Compositions compose. Plans reference the output of prior plans. The system grows in the same way the organization's operational knowledge grows — incrementally, accretively, each new capability building on the last.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hotmeshio/long-tail",
-  "version": "0.1.5",
+  "version": "0.1.7",
   "description": "Long Tail Workflows — Durable AI workflows with human-in-the-loop escalation. Powered by PostgreSQL.",
   "main": "./build/index.js",
   "types": "./build/index.d.ts",
@@ -11,14 +11,14 @@
   "files": [
     "build/",
     "docs/",
-    "lib/db/schemas/",
+    "!docs/img/",
     "dashboard/dist/",
     "LICENSE",
     "README.md"
   ],
   "scripts": {
     "clean": "rimraf ./build",
-    "build": "tsc --build tsconfig.json",
+    "build": "tsc --build tsconfig.json && cp -r lib/db/schemas build/lib/db/schemas",
     "clean-build": "npm run clean && npm run build",
     "start": "ts-node index.ts",
     "start:dev": "ts-node-dev --respawn index.ts",
@@ -61,7 +61,7 @@
   "dependencies": {
     "@anthropic-ai/sdk": "^0.82.0",
     "@aws-sdk/client-s3": "^3.1017.0",
-    "@hotmeshio/hotmesh": "^0.14.1",
+    "@hotmeshio/hotmesh": "^0.14.3",
     "@modelcontextprotocol/sdk": "^1.27.1",
     "@opentelemetry/exporter-trace-otlp-proto": "^0.215.0",
     "@opentelemetry/resources": "^2.5.1",
@@ -76,6 +76,7 @@
     "pg": "^8.13.0",
     "pino": "^9.6.0",
     "playwright": "^1.58.2",
+    "sharp": "^0.34.5",
     "socket.io": "^4.8.3",
     "zod": "^3.25.76"
   },

package/build/routes/escalations/helpers.d.ts

@@ -1,5 +0,0 @@
-/**
- * Return the role names visible to a user, or undefined if superadmin (no filter).
- * Returns an empty array when the user holds no roles at all.
- */
-export declare function getVisibleRoles(userId: string): Promise<string[] | undefined>;

package/build/routes/resolve.d.ts

@@ -1,9 +0,0 @@
-import type { Request, Response } from 'express';
-import type { ResolvedHandle } from '../services/task';
-/**
- * Resolve the (taskQueue, workflowName) pair for a workflowId.
- *
- * Looks up the task record in lt_tasks — the workflowId is enough.
- * Sends a 404 response and returns null if the workflow can't be found.
- */
-export declare function resolveHandle(req: Request, res: Response): Promise<ResolvedHandle | null>;

package/build/routes/resolve.js

@@ -1,19 +0,0 @@
-"use strict";
-Object.defineProperty(exports, "__esModule", { value: true });
-exports.resolveHandle = resolveHandle;
-const task_1 = require("../services/task");
-/**
- * Resolve the (taskQueue, workflowName) pair for a workflowId.
- *
- * Looks up the task record in lt_tasks — the workflowId is enough.
- * Sends a 404 response and returns null if the workflow can't be found.
- */
-async function resolveHandle(req, res) {
-    try {
-        return await (0, task_1.resolveWorkflowHandle)(req.params.workflowId);
-    }
-    catch (err) {
-        res.status(404).json({ error: err.message });
-        return null;
-    }
-}

package/build/routes/yaml-workflows/helpers.d.ts

@@ -1,2 +0,0 @@
-/** Return true if a Postgres error indicates an invalid/missing ID */
-export declare function isNotFoundError(err: any): boolean;

package/build/routes/yaml-workflows/helpers.js

@@ -1,8 +0,0 @@
-"use strict";
-Object.defineProperty(exports, "__esModule", { value: true });
-exports.isNotFoundError = isNotFoundError;
-/** Return true if a Postgres error indicates an invalid/missing ID */
-function isNotFoundError(err) {
-    const msg = err?.message ?? '';
-    return msg.includes('invalid input syntax for type uuid') || msg.includes('not found');
-}