@hotmeshio/long-tail 0.1.7 → 0.1.8

package/docs/epic-integration.md

@@ -1,224 +0,0 @@
-# Capturing SOPs: Epic Referral Intake
-
-## The Problem Isn't the Integration
-
-An engineering team is building for medical back-office customers who process referral intake. The customers know how to do this work. They know that BlueCross referrals in Texas require a specific prior-auth form submitted through the payer portal before the clinical documents are attached. They know that Epic flags certain `ServiceRequest` resources as urgent but the actual urgency depends on the referring provider's specialty. They know that when a coverage check returns `active` but the plan code starts with `HMO-`, there's an additional step the system doesn't surface.
-
-This knowledge is institutional. It lives in the heads of intake coordinators who've been doing this for years. Some of it is written down — in training binders, in shared docs that were last updated eighteen months ago, in Slack threads that nobody will ever find again. Most of it isn't written down at all. It's the "ask Linda" layer of operations.
-
-The engineering team doesn't need another integration platform. They need a way to capture what Linda knows and make it executable. When Linda leaves or the team scales from five coordinators to fifty, the knowledge should still be there — not as documentation that someone has to read and interpret, but as compiled tools that run the process the way Linda would.
-
-That's the actual problem. Epic's FHIR APIs are the data layer. The institutional knowledge is the value layer. Long Tail is the machine that converts one into the other.
-
----
-
-## Why Not Traditional Tools
-
-The engineering team has evaluated the usual options:
-
-**Integration middleware** builds point-to-point connections between systems. It moves data. It doesn't encode decisions. The middleware can sync a `ServiceRequest` from Epic to an internal queue, but it can't encode the rule that BlueCross-Texas referrals need a payer portal submission before document attachment. That logic ends up in custom code bolted onto the middleware, maintained by engineers who don't understand the clinical workflow, informed by specs that were wrong the day they were written.
-
-**RPA** records clicks and replays them. It captures *what* the coordinator does in the browser, not *why*. When Epic's UI changes — and it changes often — the recording breaks. When the underlying rule changes — a payer updates their prior-auth requirements — the recording doesn't know. RPA captures motion, not knowledge.
-
-**Custom software** works if the requirements are static. They aren't. Payers change rules quarterly. Epic releases updates. New customers bring new Epic configurations. The engineering team would spend most of their time maintaining the gap between what the software does and what the process requires, mediated by Jira tickets that describe the problem in terms the coordinator understands but the engineer has to translate.
-
-The team wants to bypass all of this. Connect directly to Epic's FHIR APIs. Let the people who know the process describe it. Compile what works into tools. When something changes, the tool fails, the right person fixes it, and the updated knowledge compiles back in. No translation layer between the domain expert and the executable process.
-
----
-
-## The Starting Point: Epic as Data Layer
-
-Epic exposes FHIR R4 APIs through open.epic.com. The engineering team registers a custom MCP server that wraps the FHIR endpoints their referral workflows need. Each tool maps to a FHIR operation:
-
-- `search_service_requests` — find referrals by patient, date, status, referring provider
-- `get_patient` — retrieve patient demographics by MRN or FHIR ID
-- `search_coverage` — check insurance status and plan details
-- `search_document_references` — find clinical documents linked to a referral
-- `create_task` — post a follow-up action back to Epic
-- `update_service_request` — transition referral status
-
-The server handles FHIR serialization, pagination, and query syntax. The team tags it `['ehr', 'epic', 'fhir', 'referrals']`. They also register the browser automation server for payer portals that don't have APIs.
-
-This is plumbing. It takes a week. The interesting part is what happens next.
-
----
-
-## Capturing the First SOP
-
-### The Session with Linda
-
-The engineering team sits with Linda, the senior intake coordinator. They open the Pipeline Designer — the development environment where AI-assisted exploration is allowed — and ask Linda to walk through a referral.
-
-Linda describes it as she does it:
-
-> *"First I pull up the referral in Epic and check the insurance. If it's BlueCross and the plan code starts with HMO, I have to go to the BlueCross portal and submit a prior-auth request before I can attach the clinical docs. If it's Aetna, I check whether the referring provider is in-network first, because out-of-network Aetna referrals need a different form. For everything else, I just verify the coverage is active and move to documents."*
-
-The engineer translates this into a prompt. The dynamic path runs — the LLM discovers the FHIR tools, executes the queries Linda described, follows her branching logic, and produces the result. Every FHIR call is checkpointed. The execution trace captures not just the API calls but the decision structure: the payer-specific branches, the ordering constraints, the conditions that trigger each path.
-
-Linda looks at the output. "That's right, except when the coverage check returns `active` but there's a `copay-exception` extension, you need to flag it for manual review." The engineer adds this to the prompt and reruns. The LLM adjusts.
-
-### What Just Happened
-
-Linda's institutional knowledge — the payer-specific rules, the ordering constraints, the exception handling — is now captured in an execution trace. Not in a spec document. Not in a Jira ticket. In a recorded sequence of API calls with decision points and branching logic that produced the correct result.
-
-The AI is scaffolding here. It translated Linda's description into tool calls. It won't be in the production path. But it bridged the gap between "how Linda describes the process" and "a sequence of API operations that implements it." That gap is where traditional approaches lose fidelity.
-
-### Compilation
-
-The team opens the Compilation Wizard. The compiler examines the execution trace and separates what's fixed (the branching logic, the FHIR query patterns, the payer-specific rules) from what's dynamic (the patient ID, the referral ID, the specific plan code values). It produces a deterministic DAG with typed inputs and typed outputs.
-
-The team names it `intake-referral-coverage-check`, tags it `['referrals', 'insurance', 'epic', 'intake']`, and deploys it. Linda's knowledge about payer-specific coverage rules is now a compiled, versioned, executable tool. No LLM in the execution path. It runs the process the way Linda described it, every time, without Linda.
-
----
-
-## Capturing the Next SOP, and the Next
-
-The team runs more sessions. Each one captures a different piece of the intake process:
-
-**Document verification.** Maria knows which clinical documents are required for each referral type. Orthopedic referrals need recent imaging. Cardiology needs an EKG report and recent labs. Behavioral health needs a clinical summary but not labs. Maria walks through the logic. The system captures it. The compiled tool checks `DocumentReference` resources against referral-type requirements and identifies what's missing.
-
-**Provider validation.** James knows the receiving provider network. He knows which practices have stopped accepting certain insurance plans, which have capacity constraints, and which require specific referral formats. The compiled tool queries `Practitioner` and `Organization` resources and validates the referral against James's routing knowledge.
-
-**Status transitions.** The team captures the full referral lifecycle: `draft` → `active` → `on-hold` (waiting for documents) → `active` (documents received) → `completed` (scheduled). Each transition has preconditions. Each precondition is a compiled check. The workflow won't advance a referral to `active` unless coverage is verified and required documents are attached — because that's how Linda's team actually does it.
-
-Each compiled tool encodes one person's domain knowledge as an executable, testable, versionable artifact. The SOP isn't a document anymore. It's a tool.
-
----
-
-## Connecting Directly
-
-### No Middleware in the Middle
-
-The compiled workflows call Epic's FHIR APIs directly. There's no integration platform transforming data between systems. The `ServiceRequest` resource in Epic is the referral. The `Coverage` resource is the insurance. The `Task` resource is the follow-up action. Epic is the source of truth, and the workflows read and write to it without translation.
-
-This matters for a practical reason: when Epic changes something — a new required field, a modified search parameter, an updated extension — the impact is immediate and visible. The compiled tool fails. The system creates an escalation with the full FHIR `OperationOutcome` error. The team knows exactly what changed and where.
-
-With middleware in between, the same change produces a vague data-mapping error three layers deep, and someone spends a week figuring out which system changed what.
-
-### FHIR as the Shared Language
-
-The tools the team builds speak FHIR. This means:
-
-**Epic is the canonical record.** Workflows don't maintain shadow copies of referrals in a separate database. They query Epic for current state and write results back to Epic. If an intake coordinator opens Epic directly, they see the same status the workflow set.
-
-**Tasks stay in Epic.** When a workflow identifies missing documents, it creates a FHIR `Task` resource assigned to the appropriate party. The referring provider's office sees the task in their Epic instance. No separate task-tracking system. No "check the portal for your action items" emails.
-
-**Audit is in the workflow engine.** Every FHIR call, every decision branch, every escalation is recorded in the workflow execution history. The compliance team can trace a referral from intake to scheduling, including which credentials were used, which rules were applied, and which human decisions were made along the way.
-
----
-
-## Identity: Who's Acting, On Whose Behalf
-
-The engineering team serves multiple back-office customers. Each customer connects to their own Epic instance with their own SMART on FHIR credentials (registered at open.epic.com). The same compiled workflows run against different Epic environments.
-
-**Service accounts per customer.** Each customer gets a bot account with stored Epic credentials — the SMART on FHIR private key, client ID, and token endpoint, encrypted at rest. When a workflow runs for Customer A, the credential cascade resolves Customer A's Epic token automatically.
-
-**Dual identity.** Every workflow execution records who initiated it (the scheduler, the webhook, the human operator) and who it runs as (the customer's service account). The audit trail answers both questions.
-
-**Scope enforcement.** The interceptor injects the principal's scopes into every activity. A customer scoped to `['epic:read', 'epic:referral:write']` cannot trigger a `Task` creation that requires `epic:task:write`. The FHIR token might allow it. The workflow layer doesn't.
-
-**Ephemeral credentials for payer portals.** When a workflow needs to submit a prior-auth form on a payer's web portal, it escalates to a human operator for login credentials. The credential is encrypted, stored with a TTL, exchanged atomically at dispatch time, and deleted after use. It never persists in workflow state.
-
-This isn't a section the team's customers think about. It's plumbing the engineering team builds once. But it's the plumbing that makes "one codebase, many customers, many Epic instances" possible without credential leakage or cross-tenant contamination.
-
----
-
-## The Living Part: When Knowledge Needs to Update
-
-### Payer Rule Changes
-
-Aetna updates their prior-authorization requirements for orthopedic referrals. The compiled `intake-referral-coverage-check` tool doesn't know this — it encodes last quarter's rules. Referrals that should trigger prior-auth are sailing through without it.
-
-The failure surfaces when a downstream scheduling workflow can't complete because the payer rejects the referral. The escalation includes the full context: which referral, which payer, which step failed, and the payer's rejection reason.
-
-The team brings Linda back to the Pipeline Designer. "Oh, Aetna changed this in March. Now any ortho referral with a plan code starting with `PPO-` also needs prior auth, not just the HMO plans." The engineer reruns the dynamic path with the updated logic. It succeeds. They compile. The new version deploys. The old version is still there for rollback.
-
-Linda's updated knowledge is now in the system. The SOP evolved. No spec document was rewritten. No Jira ticket went through a sprint. The compiled tool *is* the current SOP.
-
-### Epic API Changes
-
-Epic updates their FHIR implementation. A field that was optional on `ServiceRequest` is now required. The compiled tool starts returning 422 errors.
-
-The system doesn't swallow these. Non-2xx FHIR responses throw typed errors with the status code, the `OperationOutcome` body, and the request context. The interceptor catches the failure and escalates with full diagnostic information.
-
-The team uses the dynamic path in development — AI is allowed there — to explore the change against Epic's sandbox. The LLM reads the error, adjusts the query, succeeds. The team compiles the corrected pattern. Production resumes.
-
-### New Customer, Different Epic Configuration
-
-A new customer's Epic instance has a custom extension on `Coverage` resources that indicates whether a plan requires a specialist referral versus an open-access referral. The existing compiled tools don't read this extension.
-
-The team doesn't rewrite the tools. They run a dynamic session against the new customer's sandbox, discover the extension, and compile a customer-specific variant of the coverage check. They tag it `['referrals', 'insurance', 'epic', 'customer-b']`. The routing layer uses tags to match the right tool to the right customer.
-
-The core intake workflow stays the same. The coverage-check step resolves to a different compiled tool depending on which customer's service account is running. The process is the same. The payer-specific and Epic-specific details differ. The tag system handles the routing.
-
----
-
-## The Composition: SOPs Build on SOPs
-
-Individual compiled tools are building blocks. The full referral intake process composes them:
-
-1. **Validate referral** — check required FHIR fields, verify referring provider, confirm receiving practice accepts the referral type
-2. **Check coverage** — payer-specific insurance verification with prior-auth detection
-3. **Verify documents** — referral-type-specific clinical document requirements
-4. **Resolve gaps** — if documents are missing or prior-auth is needed, escalate to the appropriate party and wait
-5. **Transition status** — advance the referral through Epic's status lifecycle with appropriate `Task` creation
-
-Each step is a compiled tool that encodes one person's domain knowledge. The composition is itself compiled — a deterministic DAG that sequences the steps with conditional branches and escalation points. No LLM anywhere in the production path.
-
-The escalation in step 4 is where institutional knowledge grows. When the workflow pauses for a missing document, the coordinator who resolves it might say: "This always happens with referrals from Dr. Chen's office — they never attach the imaging report." That pattern becomes a compiled pre-check that flags Dr. Chen's referrals for document follow-up *before* the gap-resolution step. The SOP gets tighter with use.
-
----
-
-## The Browser Layer: Payer Portals Without APIs
-
-Some parts of referral intake require browser interaction. Payer authorization portals, legacy fax gateways, state Medicaid enrollment systems — these have web interfaces but no APIs.
-
-The team compiles browser workflows the same way they compile FHIR workflows. In development, the LLM navigates the portal, discovers the form fields, submits the request, and captures the confirmation. The engineer reviews the execution, corrects any brittle selectors, and compiles. In production, the compiled DAG replays the exact browser sequence — no LLM, deterministic, durable.
-
-When a portal redesigns its interface, the compiled script fails. The escalation includes a screenshot of the new layout. The team recompiles against the updated page. The browser backend is pluggable — Playwright locally, a managed browser service in production — but the compiled workflow doesn't change.
-
-This is where "bypassing traditional tools" is most literal. Instead of an RPA platform that records mouse movements and breaks on every UI change, the team has compiled browser workflows that encode the *intent* (submit prior-auth form with these fields) in a structure that can be updated when the surface changes.
-
----
-
-## What the Engineering Team Built
-
-After three months, the engineering team has delivered something that none of the traditional tools could:
-
-**Executable SOPs.** Each compiled workflow encodes a specific piece of the intake process as described by the people who do it. The SOPs aren't documents — they're running code that produces the same result the domain expert would, deterministically, across every customer's Epic instance.
-
-**A knowledge inventory that grows.** Every session with a domain expert produces compiled tools. Every escalation resolution that reveals a repeatable pattern produces another. The system accumulates operational knowledge monotonically — new compilations add capabilities, version history preserves the evolution.
-
-**Self-maintaining integrations.** When Epic's API changes, the affected tool fails and escalates with precise diagnostic context. The fix is a recompilation, not a rewrite. When a payer changes rules, the domain expert describes the change, the dynamic path validates it, and the compiled tool updates. The engineering team isn't maintaining a gap between spec and implementation because there is no spec — the compiled tool is the specification.
-
-**Customer isolation without code branches.** Service accounts, credential scoping, and tag-based routing handle multi-tenancy. The same compiled workflows serve every customer. Customer-specific variations are separate compiled tools selected by the routing layer. No `if (customer === 'A')` branches in the codebase.
-
-**A compliance-ready audit trail.** Every referral processed has a complete execution record: which FHIR calls were made, which credentials were used, which human decisions were involved, which compiled tool version ran. The data is in Postgres — queryable, exportable, backupable.
-
-### What Didn't Ship
-
-No middleware platform. No RPA licenses. No integration-platform-as-a-service subscription. No custom application with a six-month development timeline and a maintenance budget.
-
-The engineering team wrote one MCP server that wraps Epic's FHIR APIs. They compiled workflows from sessions with domain experts. They deployed it on their own infrastructure — Postgres and containers. The institutional knowledge that previously lived in Linda's head and Maria's training binder now lives in compiled tools that run without either of them present.
-
----
-
-## From 0 to 1, and 1 to 1.5
-
-The system didn't arrive complete. It grew.
-
-**0 to 1** was the first compiled SOP: Linda's coverage-check workflow running against Epic's sandbox. A single tool that encoded a single person's knowledge about a single piece of the intake process. It proved the concept — institutional knowledge could be captured as a compiled tool and executed without AI, without the domain expert present, against a real FHIR API.
-
-**1 to 1.1** was multi-tenancy: the same tool running against multiple customers' Epic instances with proper credential isolation.
-
-**1.1 to 1.2** was breadth: capturing more SOPs from more domain experts. Document verification from Maria. Provider routing from James. Each session added to the tool inventory.
-
-**1.2 to 1.3** was composition: individual tools assembled into the full intake pipeline, with escalation points where human judgment is still required.
-
-**1.3 to 1.5** is the living part: the system updating itself when the landscape shifts. A payer changes rules, a compiled tool fails, the domain expert describes the change, the tool recompiles. The gap between "how the process works" and "what the system does" stays small because the feedback loop is tight — failure to escalation to resolution to recompilation.
-
-**1.5 to 2** is where it gets interesting. The escalation history reveals patterns. Which payers change rules most often. Which Epic configurations cause the most failures. Which referral types need the most human intervention. The engineering team uses this data to prioritize: build more specific tools where the escalation rate is highest. The system's own operational history guides its evolution.
-
-The long tail of referral edge cases gets shorter. Not because someone anticipated every case, but because the system compiled solutions as the team encountered them. Each solved case is a permanent capability. The SOP library grows. The escalation rate drops. The domain experts spend less time on routine intake and more time on the genuinely novel cases — the ones that actually need human judgment.
-
-That's what a living system looks like. Not software that was correct when it shipped. Software that gets more correct the longer it runs, because it absorbs what the people around it know.

package/docs/workflow-builder.md

@@ -1,371 +0,0 @@
-# The Workflow Builder: From Prompt to Plan
-
-## The Gap Between a Prompt and a System
-
-The current workflow builder takes a prompt and produces a single DAG. Describe a task, the LLM constructs YAML, you deploy a tool. This works when the unit of work is one workflow — screenshot a page, check a patient's coverage, submit a form.
-
-But the engineering team sitting with Linda doesn't have one task. They have a PRD. A technical design document. Pages of specifications describing a referral intake system with branching paths, composed sub-processes, human review gates, and payer-specific variations. The system they need isn't one workflow. It's a set of interrelated workflows — some calling others, some waiting for human input, some branching based on data conditions — that together implement a complete operational process.
-
-Today, the engineer would decompose the PRD manually, build each workflow individually through the wizard, deploy them into the same namespace, and wire the composition by hand. That works, but it's the same translation problem the builder was designed to eliminate: the engineer becomes the bottleneck between the specification and the executable system.
-
-The evolution is plan mode. When the input exceeds what a single workflow can express — because it's long, because it describes composition, because it references multiple processes — the builder enters a planning phase. It decomposes the specification into workflows, identifies their relationships, builds them leaf-first, and deploys them as a coordinated set.
-
----
-
-## Two Entry Points, Same Destination
-
-The builder already has two modes:
-
-**Discover & Compile.** Run a dynamic execution with AI, then compile the trace into a deterministic DAG. The execution is the specification — the compiler extracts the pattern from what actually happened.
-
-**Direct Build.** Describe the workflow, and the LLM constructs the YAML directly from tool schemas without executing anything. The engineer's description is the specification.
-
-Plan mode extends Direct Build. The input is the same textarea. The difference is what the input contains. A short prompt ("check referral coverage for a patient") produces a single workflow. A long specification with composition, branching, and multiple processes triggers plan mode.
-
-The detection is straightforward: length beyond a threshold, or structural signals in the content — references to multiple steps that are themselves processes, explicit composition language ("this flow calls that flow"), mentions of planning or phased execution. When plan mode activates, the builder wraps the existing single-workflow wizard in an outer loop that plans first, then builds each workflow through the same pipeline.
-
----
-
-## What Plan Mode Produces
-
-A plan is a decomposition of the specification into workflows with defined relationships:
-
-**Leaf workflows** have no dependencies on other custom workflows. They call MCP tools directly — FHIR queries, browser actions, file operations. They're the atomic units of the system.
-
-**Composition workflows** call leaf workflows (or other composition workflows) as activities. They encode sequencing, branching, and human-in-the-loop gates. They're the orchestration layer.
-
-**The build order is leaf-first.** The planner identifies terminal workflows — the ones that don't invoke other custom workflows — and builds those first. Then it builds the workflows that compose them. Then the workflows that compose those. The tree builds from the leaves up, because each layer needs to reference the tools that already exist in the layer below.
-
-Each workflow in the plan belongs to a namespace. Workflows in the same namespace are deployed together as one HotMesh application. They can invoke each other using HotMesh's `await` activity — a direct, typed call within the same execution engine. This is the tightest composition: same deployment, same transaction boundary, same namespace.
-
-Workflows in different namespaces invoke each other as MCP tools. The calling workflow treats the target as any other tool — discovered by tag, invoked through the MCP protocol, results returned as structured data. This is the loosest composition: independent deployment, independent versioning, connected only through the tool interface.
-
-The planner decides which workflows belong together based on coupling. Workflows that share data context, that are always deployed together, that represent phases of the same process — same namespace. Workflows that represent independent capabilities, that might be reused across processes, that have different versioning cycles — separate namespaces.
-
----
-
-## The Outer Loop
-
-The existing wizard has four steps in builder mode: Describe, Profile, Deploy, Test. Plan mode wraps this in an outer loop:
-
-**Plan step.** The specification goes in. The planner produces a structured decomposition: which workflows to build, in what order, with what relationships. The dashboard displays this as a list or graph — the set of workflows, their dependencies, their namespaces. The engineer reviews, adjusts, and confirms.
-
-**Build loop.** For each workflow in the plan (leaf-first), the system runs the existing builder pipeline: construct YAML from the specification slice, validate, present for review. The engineer sees the familiar Profile → Deploy → Test sequence for each workflow, but the context is richer — the plan shows where this workflow fits in the larger system, which workflows call it, which it calls.
-
-**Verification.** After all workflows are built and deployed, the system runs the composition end-to-end. The engineer provides inputs at the top level, and the execution cascades through the composed workflows. Escalation points pause and wait. Branch conditions route correctly. The full system operates as specified.
-
-The UX for the outer loop is familiar. The plan step is a new panel above the wizard — a progress view showing which workflows have been built, which are next, which are pending. When the engineer clicks into a specific workflow, they see the standard wizard steps. When they step back out, they see the plan.
-
-Real-time updates flow through the existing event system. As each workflow compiles, deploys, and activates, the plan view updates. Build progress, deployment status, test results — all surfaced through the same SSE/NATS topic subscriptions the wizard already uses.
-
-Plan view has both expanded, collapsed modes. These serve to infrorm what is going on at a global level, but also can be collapsed sufficiently to provide high-level progress while allowing the engineer to focus on the target workflow currently loaded in the wizard.
-
----
-
-## Workflow Sets: The Data Model
-
-The current `lt_yaml_workflows` table stores individual workflows. Plan mode produces sets of related workflows. The relationship between them — which calls which, which namespace they share, which plan produced them — needs a home.
-
-A new entity: **workflow set**. A set is a named group of workflows produced by a single plan. It records:
-
-- The original specification (the PRD, the TDD, the markdown that was pasted in)
-- The plan (the decomposition into workflows with relationships)
-- The namespaces involved
-- The build order
-- The status of each workflow in the set (planned, building, deployed, active, failed)
-
-Each `lt_yaml_workflows` record gains an optional `set_id` foreign key. Workflows not produced by a plan have no set. Workflows produced by a plan reference their set, and through it, their siblings and their relationships.
-
-The set is a first-class entity in the dashboard. An engineer can view all workflows in a set, see their dependency graph, deploy or redeploy the set, and trace execution through composed workflows. When a leaf workflow is updated (because Epic changed an API, because a payer changed rules), the set view shows which composition workflows are affected.
-
-This also solves the grouping problem for workflows that aren't plan-produced. An engineer who builds three related workflows individually can group them into a set after the fact. The set is an organizational unit, not just a plan artifact.
-
----
-
-## Composition Mechanics
-
-### Same-Namespace Composition
-
-Workflows in the same namespace are deployed as a single HotMesh application. They share a deployment lifecycle — when any workflow in the namespace is updated, the entire namespace redeploys (merging all YAML graphs).
-
-Within the same namespace, a workflow invokes another using a worker activity whose `workflowName` matches the target's `subscribes` topic. The call is synchronous from the caller's perspective — the activity blocks until the child workflow completes — but durable underneath. If the worker process crashes, the child workflow's progress is preserved, and execution resumes from the last checkpoint.
-
-Data flows through typed input/output schemas. The caller maps its data to the child's input schema. The child's output maps back to the caller's next activity. The YAML wiring is explicit — every field is traced.
-
-### Cross-Namespace Composition
-
-Workflows in different namespaces are independent MCP tools. The calling workflow invokes them through the standard MCP tool protocol — the same mechanism used for browser automation, FHIR queries, or any other tool.
-
-This is composition through the tool layer. The caller doesn't know or care whether the target is a compiled workflow, a built-in MCP server, or an external service. It discovers the tool by tag, passes arguments, and receives results. The target workflow runs in its own namespace, with its own deployment lifecycle, its own versioning.
-
-Cross-namespace composition is the right choice when:
-
-- The target workflow is reusable across processes (a coverage check used by both intake and scheduling)
-- The target has a different release cadence (payer-specific tools updated independently)
-- The target belongs to a different team or domain
-
-### The Composition Spectrum
-
-Same-namespace `await` and cross-namespace MCP tool calls are two points on a spectrum. The planner chooses based on coupling signals in the specification. Tightly coupled processes that share data context and deploy together go in the same namespace. Independent capabilities that serve multiple consumers go in separate namespaces.
-
-An engineer can override the planner's choice. Move a workflow from one namespace to another, and the composition mechanism changes automatically — `await` becomes an MCP tool call, or vice versa. The workflow's logic doesn't change. Only the wiring.
-
----
-
-## The Prompt Strategy
-
-The builder's LLM prompt is the critical path. It teaches the model how to construct valid HotMesh YAML — activity definitions, transitions, `@pipe` data flow, signal hooks, iteration patterns, collision-proofing suffixes. The existing prompt handles single workflows well.
-
-Plan mode extends the prompt in two dimensions:
-
-**Decomposition.** The planner prompt takes the full specification and produces a structured plan: workflow names, descriptions, input/output contracts, dependencies, namespace assignments, build order. This is a different LLM task than YAML construction — it's architectural reasoning about how to decompose a system into workflows.
-
-**Cross-references.** When the builder constructs a composition workflow, it needs to know the input/output schemas of the workflows it calls. The prompt includes these schemas as context — "workflow `check-coverage` accepts `{patient_id, referral_id}` and returns `{covered, prior_auth_required, plan_code}`." The builder wires the composition using these contracts.
-
-The prompt strategy for complex specifications is iterative. The planner makes a first pass. The engineer reviews. Adjustments feed back into the planner. Each workflow in the plan is built with the full plan as context — the builder knows where each piece fits. This is why plan mode is a loop, not a single shot.
-
-For very large specifications — a full PRD with edge cases, error handling, and operational requirements — the planner may produce nested plans. A top-level plan decomposes into sub-systems. Each sub-system has its own namespace and its own set of workflows. The sub-systems compose through MCP tool calls. This is a plan that produces plans — the outer loop runs recursively.
-
----
-
-## Human-in-the-Loop in Compiled Workflows
-
-The existing system ships a human-queue MCP server with tools for durable pause points: `escalate_to_human`, `check_resolution`, `escalate_and_wait`. Compiled YAML workflows use these as regular activities.
-
-In the YAML, a human-in-the-loop step is two activities:
-
-1. A **worker** activity that calls `escalate_to_human` (or `escalate_and_wait`) with the escalation payload — what needs review, what context to show, what role should handle it.
-2. A **hook** activity that pauses the workflow and waits for a signal. The signal carries the human's response — approved/rejected, corrected data, additional instructions.
-
-The hook's `signal_schema` defines what the human provides. The dashboard renders a form from this schema. When the human submits, the signal fires, the hook receives the data, and the workflow resumes.
-
-Plan mode makes this explicit in the decomposition. When the specification says "a coordinator reviews the referral before scheduling," the planner creates a workflow with an escalation step between the validation activities and the scheduling activities. The escalation's role, payload, and expected response are derived from the specification.
-
-Conditional transitions after the hook encode the branching: if approved, proceed to scheduling; if rejected, escalate to the referring provider; if corrected, re-run validation with the corrected data. These branches are deterministic — the YAML encodes every path. The human's choice selects the path. No LLM involved.
-
----
-
-## What the Engineer Sees
-
-### Pasting the PRD
-
-The engineer opens the Pipeline Designer in builder mode. The textarea is the same one that accepts a short prompt. This time, the engineer pastes three pages of markdown — a referral intake specification with sections for validation, coverage checking, document requirements, payer-specific rules, escalation chains, and scheduling handoff.
-
-The system detects the input's complexity and activates plan mode. The UI transitions: above the wizard steps, a new panel appears showing the plan as it forms.
-
-### The Plan View
-
-The planner produces a decomposition. The dashboard renders it as a dependency graph:
-
-```
-referral-intake (namespace: referral-ops)
-├── validate-referral (leaf)
-├── check-coverage (leaf, payer-branching)
-├── verify-documents (leaf, referral-type-specific)
-├── resolve-gaps (composition: escalate → wait → retry)
-├── route-to-scheduling (leaf)
-└── process-referral (composition: orchestrates all above)
-```
-
-Each node shows: name, type (leaf/composition), namespace, status (planned/building/deployed). The engineer can click into any node to see the planned input/output schema and which tools it will use.
-
-The engineer reviews. "The coverage check should be in its own namespace — we reuse it for eligibility verification in another process." They drag the node to a new namespace. The plan adjusts: the composition workflow that called it now references it as an MCP tool instead of an `await` activity.
-
-### Building Leaf-First
-
-The engineer confirms the plan. The system begins building, starting with leaf workflows. The plan view updates in real time — each node's status changes from "planned" to "building" to "deployed" as the builder completes each workflow.
-
-For each workflow, the engineer can expand into the standard wizard: review the YAML, adjust the profile, deploy, test with sample inputs. Or they can let the builder proceed automatically and review the full set at the end.
-
-### Testing the Composition
-
-After all workflows deploy, the engineer runs the top-level `process-referral` workflow with a test referral. The execution cascades: validation calls the validate-referral tool, coverage check invokes the check-coverage MCP tool (cross-namespace), document verification runs in-namespace. At the escalation step, the workflow pauses. The engineer resolves it through the dashboard. The workflow resumes and completes.
-
-The plan view shows the execution trace across all workflows — a unified timeline of which tools ran in which workflow, with durations and data flow visible.
-
----
-
-## The Epic Story, Continued
-
-The engineering team building referral intake for back-office customers doesn't paste one prompt at a time. They have specifications. They've sat with Linda and Maria and James, captured the full intake process, and written it up as a PRD with sections, edge cases, and decision tables.
-
-They paste the PRD into the builder. Plan mode activates. The planner decomposes it:
-
-- **Namespace `referral-intake`**: `validate-referral`, `verify-documents`, `resolve-gaps`, `route-to-scheduling`, `process-referral` (composition)
-- **Namespace `payer-tools`**: `check-coverage-bluecross`, `check-coverage-aetna`, `check-coverage-default`, `check-coverage` (router composition)
-- **Namespace `epic-fhir`**: Low-level FHIR operations if not already registered as MCP tools
-
-The payer-specific coverage tools are in their own namespace because they version independently — when Aetna changes rules, only `check-coverage-aetna` updates. The intake workflows reference them as MCP tools.
-
-The engineer reviews the plan, adjusts namespace assignments, confirms. The builder produces the full set of workflows. The engineer tests the composition end-to-end against Epic's sandbox. Linda's knowledge, Maria's knowledge, James's knowledge — encoded in a coordinated set of deterministic tools that run without AI, without the domain experts, against any customer's Epic instance.
-
-When the PRD changes — and it will — the engineer pastes the updated section. Plan mode produces a delta: which workflows need rebuilding, which are unchanged, which have new dependencies. The system evolves incrementally, the same way the specification evolved.
-
----
-
-## Implementation Surface
-
-### Data Model
-
-**New table: `lt_workflow_sets`**
-
-| Column | Type | Purpose |
-|--------|------|---------|
-| `id` | UUID | Primary key |
-| `name` | TEXT | Human-readable set name |
-| `description` | TEXT | Set description |
-| `specification` | TEXT | Original input (PRD, TDD, markdown) |
-| `plan` | JSONB | Decomposition: workflows, relationships, namespaces, build order |
-| `namespaces` | TEXT[] | App IDs involved |
-| `status` | TEXT | `planning` / `building` / `deployed` / `active` / `failed` |
-| `source_workflow_id` | TEXT | Builder workflow that produced the plan |
-| `created_at` | TIMESTAMPTZ | |
-| `updated_at` | TIMESTAMPTZ | |
-
-**Addition to `lt_yaml_workflows`:**
-
-| Column | Type | Purpose |
-|--------|------|---------|
-| `set_id` | UUID (nullable, FK) | References `lt_workflow_sets.id` |
-| `set_role` | TEXT (nullable) | `leaf` / `composition` / `router` |
-| `set_build_order` | INTEGER (nullable) | Build sequence within the set |
-
-### API Endpoints
-
-**Plan lifecycle:**
-- `POST /api/workflow-sets` — create set from specification, triggers planning
-- `GET /api/workflow-sets` — list sets with status filters
-- `GET /api/workflow-sets/:id` — get set with plan and workflow statuses
-- `PUT /api/workflow-sets/:id/plan` — update plan (engineer adjustments)
-- `POST /api/workflow-sets/:id/build` — start building (leaf-first)
-- `POST /api/workflow-sets/:id/deploy` — deploy all namespaces
-- `GET /api/workflow-sets/:id/workflows` — list workflows in set with relationships
-
-**Extended workflow endpoints:**
-- `GET /api/yaml-workflows?set_id=:id` — filter by set membership
-
-### Workflow: `mcpWorkflowPlanner`
-
-A new system workflow that decomposes specifications into plans. Activities:
-
-- `analyzeSpecification` — detect complexity, extract structural signals
-- `decomposeIntoWorkflows` — LLM-driven decomposition with namespace assignment
-- `validatePlan` — check for circular dependencies, missing contracts, namespace conflicts
-- `refinePlan` — incorporate engineer feedback
-
-The planner workflow produces the plan. The existing `mcpWorkflowBuilder` workflow builds each individual workflow in the plan. The orchestration between planner and builder is itself a composition — the planner calls the builder for each workflow in the plan, leaf-first.
-
-### Dashboard Components
-
-**PlanView** — the outer loop panel showing the workflow dependency graph, build status, and namespace groupings. Wraps the existing wizard steps.
-
-**PlanNode** — individual workflow in the plan view. Shows name, type, namespace, status, and links to the standard wizard for that workflow.
-
-**SetListPage** — list of workflow sets, filterable by status and namespace. Entry point for managing multi-workflow systems.
-
-### Events
-
-Plan mode publishes events on the existing NATS topic space:
-
-- `lt.plan.created` — plan produced from specification
-- `lt.plan.workflow.building` — individual workflow build started
-- `lt.plan.workflow.deployed` — individual workflow deployed
-- `lt.plan.completed` — all workflows in plan built and deployed
-- `lt.plan.failed` — build failure with context
-
-The dashboard subscribes to these for real-time plan view updates.
-
----
-
-## Prompt Strategy: Teaching the LLM to Build YAML
-
-### What Exists
-
-The current builder prompt (`prompts.ts`) teaches four activity types — trigger, worker, hook, cycle — with strong coverage of `@pipe` RPN semantics and collision-proofing rules. The RPN section ("operands THEN operator") is particularly effective; LLMs reliably produce correct pipe expressions when the stack-machine framing is clear.
-
-The prompt includes an abbreviated operator list: the most-used methods from `@string`, `@date`, `@math`, `@number`, `@array`, `@object`, `@conditional`, and `@json`. This is sufficient for single workflows that transform strings, build paths, and iterate arrays.
-
-### What's Missing
-
-Three activity types are absent from the prompt: **await**, **signal**, and **interrupt**. These are precisely the types needed for composition and cross-workflow communication — the core of plan mode.
-
-The operator catalog is abbreviated. The full HotMesh pipe system exposes 13 handler categories with over 150 methods. Most are rarely needed, but gaps in the catalog produce workarounds: the LLM constructs multi-step pipes to accomplish what a single operator would do, or worse, hardcodes values that should be computed.
-
-Conditional transitions — the mechanism for branching based on activity output — are not documented in the prompt. The LLM can produce linear flows but struggles with branches, error routing, and cycle exit conditions.
-
-### The Reference System
-
-Rather than inflating the prompt with everything, the builder uses a modular reference system. Reference files live alongside the prompt at `system/workflows/mcp-workflow-builder/reference/`:
-
-**`activity-types.md`** — Complete documentation of all 8 activity types with YAML examples, properties, and usage guidance. Covers trigger, worker, hook (all 4 modes), cycle, await (sync and fire-and-forget), signal (one and all), and interrupt (self and remote).
-
-**`pipe-functions.md`** — Full catalog of every `@pipe` operator across all 13 handler categories with signatures and return types.
-
-### Injection Strategy
-
-The prompt is assembled from layers. The base prompt always includes:
-
-- Trigger, worker, hook, cycle — the types every workflow needs
-- `@pipe` RPN rules with the "common mistake" examples
-- Core operators: `@string` (concat, substring, toLowerCase), `@date` (now, yyyymmdd, toISOString), `@math` (add), `@array` (get, length), `@object` (get)
-- Construction rules (collision-proofing, _scope threading, workflowName, job.maps)
-- Clarification protocol
-
-Additional context is injected based on what the specification requires:
-
-**Composition detected** (specification references child workflows, sub-processes, or the plan contains multiple workflows) → inject the **await** activity reference. This teaches the LLM how to invoke child workflows within the same namespace, how to wire input/output contracts, and how to use fire-and-forget mode.
-
-**HITL detected** (specification mentions human review, approval gates, escalation) → inject the **signal** activity reference and the full **hook** web-hook mode documentation. This teaches signal routing through the `hooks:` section, `code: 200` vs `code: 202`, and the escalation-then-wait pattern.
-
-**Cancellation detected** (specification mentions timeout, abort, cancel) → inject the **interrupt** activity reference. Self-interrupt for validation failures, remote interrupt for cancelling child workflows.
-
-**Branching detected** (specification mentions conditional paths, error handling, routing) → inject **conditional transitions** documentation with examples of `conditions.code`, `conditions.match`, and multi-target transition arrays.
-
-**Complex transforms detected** (specification involves date arithmetic, array manipulation, object construction beyond simple field references) → inject the extended **pipe function catalog** for the relevant categories.
-
-### Why Not Include Everything
-
-Prompt length directly affects YAML quality. The current prompt is approximately 270 lines — tight enough that the LLM reads and applies every rule. Adding the full activity type reference (~250 lines) and the full pipe catalog (~300 lines) would triple the prompt size. The LLM would skim, miss construction rules, and produce more errors.
-
-The injection strategy keeps the prompt focused on what this specific workflow needs. A simple three-step linear workflow gets the base prompt. A multi-workflow composition with escalation gates gets the base prompt plus await, signal, hook web-hook mode, and conditional transitions. The prompt grows with complexity, not unconditionally.
-
-### Refinement Loop
-
-When a built workflow fails testing, the `REFINE_PROMPT` guides correction. The refinement prompt can also inject activity-type references that the initial build missed — if the failure is an incorrect await configuration, the await reference is added to the refinement context. The system learns which references each workflow type needs through the feedback loop.
-
-### Enrichments to the Existing Prompt
-
-Based on the full HotMesh documentation, these additions improve accuracy without significant length:
-
-1. **Conditional transitions.** Add a section showing multi-target transitions with `conditions`:
-   ```yaml
-   transitions:
-     check_status:
-       - to: handle_error
-         conditions:
-           code: 500
-       - to: proceed
-   ```
-   This is essential for branching workflows and currently undocumented.
-
-2. **Extended pipe operators in the abbreviated list.** Add: `@string.endsWith`, `@string.startsWith`, `@string.padStart`, `@string.replace`, `@array.join`, `@array.sort`, `@array.slice`, `@object.create`, `@object.assign`, `@object.entries`, `@object.fromEntries`, `@conditional.nullish`, `@conditional.equality`, `@conditional.strict_equality`, `@number.isInteger`, `@number.parseFloat`, `@number.parseInt`, `@number.toFixed`. These appear frequently in real workflow specifications and their absence forces workarounds.
-
-3. **Hook sleep mode.** The existing prompt mentions hook as "durable pause point" but doesn't show the sleep configuration. Add: `sleep: 5` for fixed delays, `@pipe` expression for dynamic delays.
-
-4. **`stats` on trigger.** Custom job IDs and key-based lookups are needed for idempotent workflows. Add the `stats.id` and `stats.key` pattern.
-
-5. **`await: false` pattern.** Fire-and-forget child workflows are common in composition. A one-line addition to the hook/cycle section would cover it: "For fire-and-forget child workflows, use `type: await` with `await: false`."
-
-These additions total roughly 30 lines — minimal prompt growth for significant capability gain.
-
----
-
-## What This Enables
-
-An engineer with a specification — a PRD, a TDD, a markdown document describing a multi-step operational process — pastes it into the builder and gets back a coordinated set of deterministic workflows. Leaf workflows call MCP tools. Composition workflows orchestrate leaves. Human-in-the-loop gates pause for review. Payer-specific branches route correctly. The full system deploys as a set of tools that other workflows can compose further.
-
-The specification is the input. The compiled DAGs are the output. Plan mode is the compiler that bridges the gap. The engineer reviews and adjusts at every step — this isn't autonomous generation. It's assisted decomposition, where the LLM handles the YAML construction and the engineer handles the architectural decisions.
-
-The citizen developer illusion dissolves. What remains is an engineer who understands the domain, understands composition, and has a tool that translates specifications into executable workflows without writing YAML by hand. The tool handles the syntax. The engineer handles the semantics.
-
-And because every compiled workflow is an MCP tool, the output of one plan can be the input of another. A set of referral intake workflows becomes a tool that a scheduling system calls. A set of payer verification workflows becomes a tool that an eligibility checker calls. Compositions compose. Plans reference the output of prior plans. The system grows in the same way the organization's operational knowledge grows — incrementally, accretively, each new capability building on the last.