@hotmeshio/long-tail 0.1.4 → 0.1.5

package/docs/api/yaml-workflows.md

@@ -0,0 +1,312 @@
+# YAML Workflows API (Process Servers)
+
+Manage YAML workflow definitions — the compiled deterministic pipelines generated from successful MCP triage executions. Each namespace (`app_id`) acts as a process server; each workflow is a tool within that server. All endpoints require authentication.
+
+## List workflows
+
+```
+GET /api/yaml-workflows
+```
+
+**Query parameters:**
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `status` | `string` | Filter by status: `draft`, `deployed`, `active`, `archived` |
+| `graph_topic` | `string` | Filter by graph topic (tool name) |
+| `app_id` | `string` | Filter by namespace/server (exact match) |
+| `search` | `string` | Search name, graph topic, description, or app ID (case-insensitive) |
+| `source_workflow_id` | `string` | Filter by source execution workflow ID |
+| `limit` | `integer` | Max results (default: 50) |
+| `offset` | `integer` | Pagination offset (default: 0) |
+
+**Response 200:**
+
+```json
+{
+  "workflows": [
+    {
+      "id": "a1b2c3d4-...",
+      "name": "rotate-and-verify",
+      "description": "Rotate document, extract info, validate",
+      "app_id": "lt-yaml",
+      "app_version": "3",
+      "source_workflow_id": "mcp-triage-abc123",
+      "source_workflow_type": "mcpTriage",
+      "yaml_content": "app:\n  id: lt-yaml\n  version: \"3\"\n...",
+      "graph_topic": "rotate_and_verify",
+      "input_schema": { "type": "object", "properties": { "image_ref": { "type": "string" } } },
+      "output_schema": {},
+      "activity_manifest": [
+        {
+          "activity_id": "a0",
+          "title": "Rotate Page",
+          "type": "worker",
+          "topic": "rotate_and_verify/a0",
+          "tool_source": "mcp",
+          "mcp_server_id": "srv-1",
+          "mcp_tool_name": "rotate_image",
+          "input_mappings": {},
+          "output_fields": ["rotated_url"]
+        }
+      ],
+      "status": "active",
+      "deployed_at": "2025-01-10T00:00:00Z",
+      "activated_at": "2025-01-10T00:01:00Z",
+      "metadata": null,
+      "created_at": "2025-01-09T00:00:00Z",
+      "updated_at": "2025-01-10T00:01:00Z"
+    }
+  ],
+  "total": 1
+}
+```
+
+## Create a workflow (from execution)
+
+```
+POST /api/yaml-workflows
+```
+
+Generates a YAML workflow from a completed MCP triage execution.
+
+**Request body:**
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `workflow_id` | `string` | Yes | Source execution workflow ID |
+| `task_queue` | `string` | Yes | Task queue of the source execution |
+| `workflow_name` | `string` | Yes | Source workflow type name |
+| `name` | `string` | Yes | Name for the new workflow |
+| `description` | `string` | No | Human-readable description |
+| `app_id` | `string` | No | Namespace/app ID (default: auto-generated) |
+| `subscribes` | `string` | No | Topic subscription |
+| `tags` | `string[]` | No | Tags for workflow discovery |
+| `compilation_feedback` | `string` | No | Additional instructions for the compilation pipeline |
+
+**Response 201:** Created workflow record.
+
+**Response 400:** Missing required fields.
+
+**Response 409:** Workflow name already exists.
+
+**Response 422:** Source execution exhausted its tool rounds (incomplete trace).
+
+## Get a workflow
+
+```
+GET /api/yaml-workflows/:id
+```
+
+**Response 200:** Workflow record.
+
+**Response 404:** Workflow not found.
+
+## Update a workflow
+
+```
+PUT /api/yaml-workflows/:id
+```
+
+**Request body:** Partial update — `yaml_content`, `name`, `description`.
+
+**Response 200:** Updated workflow record.
+
+**Response 404:** Workflow not found.
+
+## Delete a workflow
+
+```
+DELETE /api/yaml-workflows/:id
+```
+
+Only allowed for `draft` or `archived` workflows.
+
+**Response 200:**
+
+```json
+{ "deleted": true }
+```
+
+**Response 400:** Cannot delete active or deployed workflows.
+
+**Response 404:** Workflow not found.
+
+## Deploy a workflow
+
+```
+POST /api/yaml-workflows/:id/deploy
+```
+
+Deploys all YAML workflows sharing this workflow's `app_id` as a merged version. Bumps the version, marks all non-archived siblings as `active`, and registers HotMesh workers for the deployed workflows.
+
+**Response 200:** Updated workflow record (with new version).
+
+**Response 404:** Workflow not found.
+
+## Activate a workflow
+
+```
+POST /api/yaml-workflows/:id/activate
+```
+
+Activates the deployed version and registers workers for all workflows in the same `app_id`. Requires status `deployed` or `active`.
+
+**Response 200:** Updated workflow record.
+
+**Response 400:** Workflow must be deployed first.
+
+**Response 404:** Workflow not found.
+
+## Invoke a workflow
+
+```
+POST /api/yaml-workflows/:id/invoke
+```
+
+Invoke an active YAML workflow with parameters.
+
+**Request body:**
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `data` | `object` | No | Input data matching the workflow's `input_schema` |
+| `sync` | `boolean` | No | Wait for result (default: false) |
+| `timeout` | `integer` | No | Timeout in ms for sync invocations (default: 120000) |
+
+**Async response 200:**
+
+```json
+{ "job_id": "hmsh:lt-yaml:j:abc123..." }
+```
+
+**Sync response 200:**
+
+```json
+{
+  "job_id": "hmsh:lt-yaml:j:abc123...",
+  "result": {
+    "metadata": { "jid": "abc123", "tpc": "rotate_and_verify" },
+    "data": { "rotated_url": "https://...", "verified": true }
+  }
+}
+```
+
+**Response 400:** Workflow must be active to invoke.
+
+**Response 404:** Workflow not found.
+
+## Regenerate a workflow
+
+```
+POST /api/yaml-workflows/:id/regenerate
+```
+
+Re-generate the YAML from the original source execution. Allowed for `draft`, `deployed`, and `active` workflows (not `archived`).
+
+**Request body:**
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `task_queue` | `string` | No | Override task queue (default: from source) |
+| `compilation_feedback` | `string` | No | Additional instructions for the regeneration |
+
+**Response 200:** Updated workflow record.
+
+**Response 400:** Archived workflows cannot be regenerated.
+
+**Response 404:** Workflow not found.
+
+## Archive a workflow
+
+```
+POST /api/yaml-workflows/:id/archive
+```
+
+Stops accepting invocations. If active, stops the engine.
+
+**Response 200:** Updated workflow record with status `archived`.
+
+**Response 404:** Workflow not found.
+
+## List app IDs
+
+```
+GET /api/yaml-workflows/app-ids
+```
+
+Returns distinct `app_id` values from non-archived workflows.
+
+**Response 200:**
+
+```json
+{
+  "app_ids": ["lt-yaml", "my-custom-app"]
+}
+```
+
+## Get version history
+
+```
+GET /api/yaml-workflows/:id/versions
+```
+
+Returns version history for a YAML workflow.
+
+**Query parameters:**
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `limit` | `integer` | Max results (default: 20) |
+| `offset` | `integer` | Pagination offset (default: 0) |
+
+**Response 200:** Version history records.
+
+**Response 404:** Workflow not found.
+
+## Get version snapshot
+
+```
+GET /api/yaml-workflows/:id/versions/:version
+```
+
+Returns a single version snapshot with YAML content, schemas, and activity manifest.
+
+**Path parameters:**
+
+| Parameter | Description |
+|-----------|-------------|
+| `id` | Workflow UUID |
+| `version` | Version number (positive integer) |
+
+**Response 200:** Version snapshot object.
+
+**Response 400:**
+
+```json
+{ "error": "Invalid version number" }
+```
+
+**Response 404:**
+
+```json
+{ "error": "Version 3 not found" }
+```
+
+## Get raw YAML
+
+```
+GET /api/yaml-workflows/:id/yaml
+```
+
+Returns the raw YAML content as `text/yaml`. Supports an optional `version` query parameter to retrieve YAML for a specific version.
+
+**Query parameters:**
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `version` | `integer` | Return YAML for this version instead of current |
+
+**Response 200:** YAML content (Content-Type: text/yaml).
+
+**Response 404:** Workflow or version not found.

package/docs/architecture.md

@@ -0,0 +1,221 @@
+# Architecture
+
+This document describes how Long Tail organizes workflows, activities, and MCP servers. It covers the three-tier project structure, the convention that makes activities and tools interchangeable, the system capabilities that ship by default, the built-in MCP servers, and how to register your own code. For a higher-level overview and quick start guide, see the [README](../README.md).
+
+## Contents
+
+- [Project Structure](#project-structure)
+- [The Convention](#the-convention)
+- [Three Workflow Types](#three-workflow-types)
+- [System Capabilities](#system-capabilities)
+  - [Tag-Based Tool Discovery](#tag-based-tool-discovery)
+  - [System Workflows](#system-workflows)
+- [Built-in MCP Servers](#built-in-mcp-servers)
+- [Registering Your Own](#registering-your-own)
+
+## Project Structure
+
+Long Tail organizes workflows, activities, and MCP servers into three tiers that follow the same conventions:
+
+```
+system/          Always ships. The built-in tool inventory.
+├── activities/    Vision, DB queries, HTTP fetch, file storage, ...
+├── workflows/     mcpQuery pipeline, mcpTriage pipeline
+└── mcp-servers/   Built-in MCP servers wrapping the activities
+
+examples/        Opt-in demos. Seed with `examples: true`.
+├── workflows/     review-content, verify-document, process-claim, ...
+└── types/
+
+your-app/        Your workflows. Same conventions, your directory.
+├── activities/    Your activity functions (= your MCP tools)
+├── workflows/     Your workflow functions
+└── index.ts       Barrel export of workers[]
+```
+
+**System** workflows and tools are always available. Two 3-tier pipelines handle all dynamic work: **mcpQuery** (router → deterministic | dynamic) fulfills arbitrary requests using MCP tools, and **mcpTriage** (router → deterministic | dynamic) handles AI-assisted remediation when humans can't resolve an escalation. Each pipeline has a router that checks for compiled workflows before falling back to a dynamic agentic loop.
+
+**Examples** demonstrate the patterns. They seed the dashboard with working scenarios on first run.
+
+**Your code** follows the same structure. Register workers via `start()`:
+
+```typescript
+import { start } from '@hotmeshio/long-tail';
+import { myWorkers } from './my-app';
+
+const lt = await start({
+  database: { connectionString: process.env.DATABASE_URL },
+  workers: [...myWorkers],
+  examples: true,
+});
+```
+
+All three tiers follow the same directory conventions. The next section explains that convention and why it matters.
+
+## The Convention
+
+Every tier follows the same structure. Activities are where side effects live. Workflows are deterministic functions that call activities. MCP servers are thin wrappers that expose activities as tools.
+
+```
+<dir>/
+├── activities/       Side-effect functions — each one IS an MCP tool
+│   └── analyzer.ts   export async function classify(args) { ... }
+├── workflows/
+│   └── my-flow/
+│       ├── index.ts       Workflow function (deterministic)
+│       └── orchestrator.ts  executeLT() wrapper for composition
+└── mcp-servers/
+    └── analyzer.ts    Wraps activities/analyzer.ts as an MCP server
+```
+
+The key insight: **activities are tools and tools are activities**. Write a function that classifies a document. Register it as an activity in your workflow. The same function, wrapped in an MCP server, becomes a tool that any LLM agent can call. When the triage agent uses your tool to fix an edge case, the successful sequence gets compiled into a deterministic workflow — and that workflow becomes a new tool.
+
+```typescript
+// activities/analyzer.ts — this function IS a tool
+export async function classify(args: { content: string }) {
+  const response = await llm.analyze(args.content);
+  return { category: response.category, confidence: response.confidence };
+}
+
+// workflows/classify-content/index.ts — calls the activity
+const { classify } = Durable.workflow.proxyActivities<typeof activities>({
+  activities,
+});
+
+export async function classifyContent(envelope: LTEnvelope) {
+  const result = await classify(envelope.data);
+  if (result.confidence >= 0.85) {
+    return { type: 'return', data: result };
+  }
+  return { type: 'escalation', data: result, message: 'Low confidence', role: 'reviewer' };
+}
+
+// mcp-servers/analyzer.ts — exposes the activity as a tool
+server.tool('classify', schema, async (args) => activities.classify(args));
+```
+
+With this convention in place, the system ships with a set of built-in capabilities that demonstrate the pattern.
+
+## Three Workflow Types
+
+Long Tail has three distinct workflow types. The dashboard's **Workflow Registry** page shows each with a visual badge.
+
+### Durable
+
+Any function registered as a HotMesh worker is durable. This is the baseline tier:
+
+- **Checkpointed execution** — activity results are persisted in PostgreSQL; crashes resume from the last checkpoint.
+- **Automatic retries** — failed activities retry according to their retry policy.
+- **IAM context** — the caller's identity propagates through the execution envelope.
+
+A durable workflow has no entry in `lt_config_workflows`. It runs, but without interceptor tracking, escalation chains, or task records. In the dashboard, durable workflows display a **Workflow** icon with a "Durable" label.
+
+### Certified
+
+A certified workflow is durable plus the full Long Tail control plane. It has an entry in `lt_config_workflows`, which activates:
+
+- **Interceptor wrapping** — every execution is tracked as a task in `lt_tasks` with full audit trail.
+- **Escalation chains** — returning `{ type: 'escalation' }` creates a reviewable record, routes to the correct role, and triggers re-runs on resolution.
+- **Never-fail guarantee** — unhandled errors are caught and surfaced as error escalations.
+- **Invocation controls** — `invocable: true` exposes the workflow for external invocation via the API and dashboard.
+- **Execution identity** — roles, default assignees, and `execute_as` overrides are defined in the config.
+
+In the dashboard, certified workflows display a **ShieldCheck** icon with a "Certified" label in accent blue. Promote a durable workflow to certified by creating a config entry; de-certify by removing it. The workflow code does not change.
+
+### Pipeline
+
+A pipeline workflow is a compiled deterministic workflow generated from a successful dynamic execution. The compilation pipeline extracts the tool-call sequence from an `mcpQuery` or `mcpTriage` run and produces a YAML DAG that replays the same steps without an LLM.
+
+- **No LLM per step** — the DAG executes tool calls directly with pre-wired data flow.
+- **Discoverable as a tool** — deployed pipelines become MCP tools that any workflow or agent can invoke.
+- **Automatic routing** — the `mcpQueryRouter` and `mcpTriageRouter` discover compiled pipelines via full-text search and tag matching, routing requests to the deterministic path when confidence is high.
+
+Pipeline workflows are stored in `lt_yaml_workflows` with status lifecycle: `draft` → `deployed` → `active` → `archived`. In the dashboard, pipeline workflows display a **Wand2** (magic wand) icon in purple.
+
+See the [Compilation Pipeline](compilation.md) guide for the full lifecycle and the [Workflows Guide](workflows.md) for detailed coverage of all three types.
+
+## System Capabilities
+
+### Tag-Based Tool Discovery
+
+Every MCP server is tagged with categories (`database`, `vision`, `http`, `storage`, etc.). Workflows discover tools by tag rather than hard-coding server names:
+
+```typescript
+// mcpQuery discovers all tools (or filters by user-provided tags)
+const servers = await findServersByTags(tags, 'any');
+
+// scoped discovery — only database-tagged tools
+const servers = await findServersByTags(['database', 'analytics'], 'any');
+```
+
+Tags are GIN-indexed in PostgreSQL for fast lookup. Register a new MCP server tagged `analytics` and mcpQuery automatically picks up its tools. Register one tagged `vision` and mcpTriage can use it for remediation. The tool inventory grows without code changes.
+
+### System Workflows
+
+| Pipeline | Workflows | Purpose |
+|----------|-----------|---------|
+| **mcpQuery** | `mcpQueryRouter` → `mcpDeterministic` \| `mcpQuery` | General-purpose MCP orchestration. Router discovers compiled workflows; routes to deterministic (if match) or dynamic agentic loop. |
+| **mcpTriage** | `mcpTriageRouter` → `mcpTriageDeterministic` \| `mcpTriage` | Escalation remediation. Same 3-tier pattern — router checks for compiled solutions before falling back to dynamic triage. |
+
+**mcpQuery** is the general-purpose entry point. Ask it to "take a screenshot of example.com" and it uses Playwright tools. Ask it to "fetch the latest exchange rates and save to a file" and it chains HTTP fetch with file storage. Once a useful sequence is discovered, it can be compiled into a deterministic workflow and deployed as a new tool.
+
+Engineers can register external servers — Playwright for browser automation, file storage, custom APIs. The triage agent can even recommend that engineers add a new MCP server when it encounters a capability gap. Every server's tools become available as proxy activities in deterministic workflows and as callable tools in dynamic triage.
+
+The built-in servers are the starting inventory.
+
+## Built-in MCP Servers
+
+Long Tail ships with the following built-in servers:
+
+| Server | Purpose |
+|--------|---------|
+| `long-tail-db-query` | Query tasks, escalations, process summaries, system health |
+| `long-tail-human-queue` | Route work to humans, check resolution, claim/resolve, escalate-and-wait |
+| `mcp-workflows-longtail` | Discover and invoke compiled workflow tools |
+| `long-tail-workflow-compiler` | Convert executions to YAML, deploy, list |
+| `long-tail-translation` | Translate content between languages |
+| `long-tail-vision` | Analyze and describe images using LLM vision |
+| `long-tail-playwright` | Low-level browser automation |
+| `long-tail-playwright-cli` | High-level browser automation |
+| `long-tail-docs` | Search and read product documentation |
+| `long-tail-file-storage` | Read, write, list, and delete files |
+| `long-tail-http-fetch` | HTTP requests, JSON/text fetch |
+| `long-tail-oauth` | OAuth token management |
+| `long-tail-claude-code` | Agentic coding via Claude Code CLI |
+
+Each server can provide `compile_hints` — tool-specific constraints stored in the database that guide the compilation pipeline when converting dynamic executions into deterministic workflows.
+
+For details on managing servers via the REST API or at startup, see [docs/mcp.md](mcp.md).
+
+To add your own tools to this inventory, follow the same convention.
+
+## Registering Your Own
+
+Follow the same convention as `system/` and `examples/`. Create a barrel export of your workers:
+
+```typescript
+// my-app/index.ts
+import * as myWorkflow from './workflows/my-workflow';
+import * as myOrch from './workflows/my-workflow/orchestrator';
+
+export const myWorkers = [
+  { taskQueue: 'my-queue', workflow: myWorkflow.myWorkflow },
+  { taskQueue: 'my-queue-orch', workflow: myOrch.myWorkflowOrchestrator },
+];
+```
+
+Register at startup:
+
+```typescript
+import { start } from '@hotmeshio/long-tail';
+import { myWorkers } from './my-app';
+
+const lt = await start({
+  database: { connectionString: process.env.DATABASE_URL },
+  workers: [...myWorkers],
+  mcp: { server: { enabled: true } },
+  escalation: { strategy: 'mcp' },
+});
+```
+
+Your activities become tools. Your tools become available to triage. Triage solutions become compiled workflows. Compiled workflows become tools. The cycle continues.