sea-dev 1.0.0

package/docs/agent-templates-and-runs.md

@@ -0,0 +1,859 @@
+# Agent Templates and Runs System
+
+## Overview
+
+The Agent Templates and Runs system provides a way to create reusable, multi-step AI workflows that transform data from one form structure to another. Templates define a sequence of execution steps, and runs represent individual executions of those templates against specific input submissions.
+
+## Database Schema
+
+### Core Tables
+
+#### `agent_template`
+Defines a reusable template for agent workflows.
+
+- `id`: Unique identifier
+- `organization_id`: Organization owner
+- `name`: Template name
+- `description`: Template description
+- `input_type`: Type of input (default: "manual", options: "submission", "deal", "form", "manual")
+- `input_form_id`: Form schema for template input data (required) - defines expected input structure for all runs
+- `output_type`: Type of output (default: "artifact", options: "artifact", "submission_update", "report")
+- `is_system`: Whether this is a system template (default: false)
+- `is_active`: Whether this template is active (default: true)
+- `version`: Current version string (default: "1")
+- `metadata`: Additional metadata (JSONB)
+- `deleted_at`, `created_at`, `updated_at`: Standard timestamps
+
+**Indexes:**
+- `organization_id`
+- `is_active`
+- `input_form_id`
+
+#### `agent_template_step`
+Defines individual steps within a template.
+
+- `id`: Unique identifier
+- `template_id`: Foreign key to `agent_template`
+- `parent_step_id`: Foreign key to parent step (nullable) - creates step dependency chain
+- `step_key`: Unique key for this step within the template
+- `step_order`: Execution order (integer)
+- `name`: Step name
+- `description`: Step description
+- `prompt`: LLM prompt for this step
+- `output_key`: Key for the output data
+- `output_form_id`: Form schema for output data - defines what this step produces
+- `tools_allowed`: Array of tool names allowed for this step (empty = all tools allowed)
+- `retry_policy`: Retry configuration (JSONB) - not yet implemented
+- `created_at`, `updated_at`: Standard timestamps
+
+**Indexes:**
+- `template_id`
+- `parent_step_id`
+- `output_form_id`
+- `(template_id, step_order)` - for ordered retrieval
+- Unique: `(template_id, step_key)`
+
+**Note on Input Handling:**
+- Steps do not define their own input forms
+- First step uses template's `input_form_id`
+- Subsequent steps use parent step's `output_form_id` as input
+- This creates a natural data flow: template input → step 1 output → step 2 output → ...
+
+#### `agent_template_version`
+Stores template definitions for versioning (planned, not fully implemented).
+
+- `id`: Unique identifier
+- `template_id`: Foreign key to `agent_template`
+- `version`: Version string
+- `definition`: Full template definition snapshot (JSONB)
+- `created_at`, `updated_at`: Standard timestamps
+
+**Indexes:**
+- `template_id`
+- Unique: `(template_id, version)`
+
+**Note:** Template versioning is planned but not yet implemented in the UI. The intended design is:
+- Runs would link to a specific template version
+- Each template would have an "active version" pointer
+- This allows templates to evolve without breaking historical runs
+
+#### `agent_run`
+Represents a single execution of a template.
+
+- `id`: Unique identifier
+- `organization_id`: Organization owner
+- `conversation_id`: Associated conversation
+- `template_id`: Foreign key to `agent_template` (nullable if template deleted)
+- `template_version_id`: Foreign key to `agent_template_version` (nullable)
+- `status`: Run status (default: "pending", options: "pending", "running", "completed", "failed", "cancelled")
+- `input_ref_type`: Type of input reference (default: "manual", options: "submission", "deal", "form", "manual")
+- `input_ref_id`: ID of the input reference (nullable)
+- `trigger_type`: How the run was triggered (default: "manual")
+- `error_summary`: Error message if failed (nullable)
+- `started_at`: When execution started
+- `completed_at`: When execution completed (nullable)
+- `created_at`, `updated_at`: Standard timestamps
+
+**Indexes:**
+- `organization_id`
+- `conversation_id`
+- `status`
+- `(input_ref_type, input_ref_id)`
+
+#### `agent_run_step`
+Represents the execution of a single step within a run.
+
+- `id`: Unique identifier
+- `run_id`: Foreign key to `agent_run`
+- `parent_step_id`: Foreign key to parent step in this run (nullable) - tracks step execution chain
+- `step_key`: Step identifier
+- `step_order`: Execution order
+- `name`: Step name
+- `status`: Step status (default: "pending", options: "pending", "running", "completed", "failed", "skipped")
+- `attempt`: Retry attempt number (default: 1)
+- `output_submission_id`: Output submission created by this step (nullable)
+- `reasoning`: LLM reasoning output (nullable)
+- `input`: Input data/config for this step (JSONB) - stores step configuration, not actual input data
+- `output`: Output data from this step (JSONB) - stores structured output result
+- `error`: Error message if failed (nullable)
+- `started_at`: When step started
+- `completed_at`: When step completed (nullable)
+- `duration_ms`: Execution duration in milliseconds (nullable)
+- `created_at`, `updated_at`: Standard timestamps
+
+**Indexes:**
+- `run_id`
+- `parent_step_id`
+- `output_submission_id`
+- `(run_id, status)`
+- Unique: `(run_id, step_key)`
+
+**Note on Input Submissions:**
+- Input submissions are not stored directly on run steps
+- Instead, derived from parent step's `output_submission_id`
+- For first step, input submission comes from `agent_run.input_ref_id`
+- This maintains referential integrity through the parent-child chain
+
+#### `agent_tool_call`
+Logs individual tool calls made during step execution.
+
+- `id`: Unique identifier
+- `run_step_id`: Foreign key to `agent_run_step`
+- `tool_name`: Name of the tool called
+- `args`: Tool arguments (JSONB)
+- `result`: Tool result (JSONB)
+- `status`: Call status (default: "completed")
+- `latency_ms`: Tool call latency in milliseconds (nullable)
+- `called_at`: When the tool was called
+- `created_at`, `updated_at`: Standard timestamps
+
+**Indexes:**
+- `run_step_id`
+- `tool_name`
+
+#### `agent_transformation`
+Records data transformations applied during step execution.
+
+- `id`: Unique identifier
+- `run_step_id`: Foreign key to `agent_run_step`
+- `kind`: Type of transformation (default: "map", options: "copy", "map", "formula", "aggregate", "classify", "normalize", "llm_extract")
+- `expression`: Transformation expression (nullable)
+- `metadata`: Additional metadata (JSONB)
+- `created_at`, `updated_at`: Standard timestamps
+
+**Indexes:**
+- `run_step_id`
+
+#### `agent_artifact`
+Stores artifacts (charts, tables, dashboards, markdown) produced by runs.
+
+- `id`: Unique identifier
+- `run_id`: Foreign key to `agent_run` (nullable)
+- `run_step_id`: Foreign key to `agent_run_step` (nullable)
+- `conversation_id`: Associated conversation
+- `type`: Artifact type (default: "markdown", options: "markdown", "chart", "table", "dashboard")
+- `title`: Artifact title
+- `content`: Artifact content (JSONB)
+- `metadata`: Additional metadata (JSONB)
+- `created_at`, `updated_at`: Standard timestamps
+
+**Indexes:**
+- `conversation_id`
+- `run_id`
+- `run_step_id`
+
+## User Interface
+
+### Main Agent Page (`/agents`)
+
+**Location:** [apps/main/src/routes/_authed/_app/agents/index.tsx](../apps/main/src/routes/_authed/_app/agents/index.tsx)
+
+Lists all agent conversations for the organization. Shows:
+- Conversation name (or "New Agent" if unnamed)
+- Last updated time
+- Delete button (appears on hover)
+
+**Features:**
+- "New Agent" button to create a new agent conversation
+- Links to individual agent pages
+
+### Agent Detail Page (`/agents/$agentId`)
+
+**Location:** [apps/main/src/routes/_authed/_app/agents/$agentId/index.tsx](../apps/main/src/routes/_authed/_app/agents/$agentId/index.tsx)
+
+Main chat interface for interacting with the agent.
+
+**Layout:**
+- **Left side:** Chat messages with document upload capability
+- **Right sidebar:** Two collapsible sections:
+  - **Context:** Shows relevant items (deals, forms, submissions, workflows, documents) with ability to add/remove
+  - **Artifacts:** Displays generated artifacts (charts, tables, markdown) with delete functionality
+
+**Features:**
+- Rename conversation (click on title)
+- Agent history menu (switch between agent conversations)
+- Agent actions menu (model selection, etc.)
+- Create new agent
+- Auto-expand context/artifacts panels when items are added
+
+### Agent Configuration Page (`/agents/$agentId/config`)
+
+**Location:** [apps/main/src/routes/_authed/_app/agents/$agentId/config.tsx](../apps/main/src/routes/_authed/_app/agents/$agentId/config.tsx)
+
+Three-tab interface for managing templates and testing runs.
+
+#### Tab 1: Create Template
+
+**Purpose:** Conversational interface for creating new agent templates
+
+**Features:**
+- Chat-based template creation
+- User describes the template (e.g., "Create a template that takes financial data and produces a risk assessment")
+- LLM assists in generating the template structure
+- Creates templates via `createAgentTemplateFn`
+
+**Current Implementation:**
+- Uses `ChatProvider` with `chatType="template"`
+- Connected to conversation messages
+- Invalidates router on input blur to refresh data
+
+#### Tab 2: Edit Templates & Tools
+
+**Purpose:** Edit existing templates, steps, and tool configurations
+
+**Features:**
+
+**Templates Section:**
+- Accordion view of all templates
+- Per-template editing:
+  - Template name and description
+  - Save button (appears only when changes detected)
+- Per-step editing:
+  - Step name, description, and prompt
+  - Input form selection (dropdown)
+  - Output form selection (dropdown)
+  - Tools allowed (currently read-only - **TODO**)
+  - Save button (appears only when changes detected)
+
+**Tools Section:**
+- Accordion view of all available tools
+- Per-tool editing:
+  - Shows default description
+  - Custom description textarea
+  - "Save" button to update
+  - "Clear" button to revert to default (only shown if custom description exists)
+  - Collapsible input schema viewer (JSON)
+- Badge indicator for tools with custom descriptions
+
+**Implementation Details:**
+- Local state management for edits (`templateEdits`, `stepEdits`, `descriptions`)
+- Optimistic UI updates
+- Router invalidation after saves
+- Tracks saving state per item to prevent double-saves
+
+#### Tab 3: Playground
+
+**Location:** [apps/main/src/routes/_authed/_app/agents/-components/agent-runtime-playground.tsx](../apps/main/src/routes/_authed/_app/agents/-components/agent-runtime-playground.tsx)
+
+**Purpose:** Test template execution against submissions and inspect results
+
+**Features:**
+
+**Execution Panel (Top):**
+- Template selection dropdown
+- Input submissions selector:
+  - Grouped by form (when template requires multiple input forms)
+  - Checkbox selection (supports multiple submissions)
+  - Shows submission name or truncated ID
+  - Badge display of selected submissions with remove buttons
+- "Run" button (executes template)
+- "Refresh" button (reloads run list)
+- Error display alert
+
+**Final Output Display:**
+- Shows when a run completes successfully
+- Displays final step output submission link
+- Shows output data preview (expandable JSON)
+- "View Final Submission" link
+
+**Recent Runs Panel (Bottom Left):**
+- List of recent runs (up to 50, configurable)
+- Status filter dropdown (All, Completed, Failed, Running, Pending)
+- Per-run display:
+  - Truncated run ID (first 8 chars)
+  - Status badge (color-coded)
+  - Relative timestamp
+- Click to load timeline details
+- Selected run highlighted
+
+**Run Timeline Panel (Bottom Right):**
+- Displays detailed step-by-step execution view
+- Per-step display:
+  - Status badge
+  - Step name
+  - Input submission link(s) - supports multiple
+  - Output submission link
+  - Output data preview (collapsible JSON)
+  - Reasoning text
+  - Tool calls (collapsible list with latency)
+  - Error messages (if failed)
+- Artifacts section (if any produced):
+  - Artifact title, type badge
+  - Content viewer (collapsible JSON)
+- Empty state when no run selected
+
+**Flow:**
+1. User selects template
+2. UI shows eligible submissions grouped by required form(s)
+3. User selects one or more submissions via checkboxes
+4. User clicks "Run"
+5. System executes template via `executeAgentTemplateFn`
+6. Run appears in "Recent Runs" list
+7. Timeline auto-loads for the new run
+8. User can click any run to inspect its timeline
+
+### Template Creator Component
+
+**Location:** [apps/main/src/routes/_authed/_app/agents/-components/agent-template-creator.tsx](../apps/main/src/routes/_authed/_app/agents/-components/agent-template-creator.tsx)
+
+**Purpose:** Form-based template creation (alternative to chat interface - not currently used in main UI)
+
+**Features:**
+- Template name input
+- Template description textarea
+- Step 1 prompt textarea
+- Input form dropdown
+- Output form dropdown
+- "Create Template" button
+- Error display
+
+**Note:** This component exists but is not currently integrated into the main config page. The chat-based creation (Tab 1) is the primary interface.
+
+## Data Flow
+
+### Template Creation Flow
+
+**Via Chat Interface (Tab 1):**
+1. User sends message describing desired template
+2. System processes via chat endpoint
+3. LLM generates template structure
+4. `createAgentTemplateFn` creates:
+   - `agent_template` record
+   - `agent_template_step` record(s)
+   - `agent_template_version` record (with full definition)
+5. Conversation's `activeTemplateId` and `executionMode` updated to "task"
+6. Router invalidates, UI refreshes
+
+**Via Direct Form (AgentTemplateCreator component):**
+1. User fills form fields
+2. `createAgentTemplateFn` called with:
+   - `conversationId`
+   - `name`, `description`
+   - `inputFormId`, `outputFormId`
+   - `stepPrompt`
+3. Creates single-step template in transaction:
+   - `agent_template` with metadata `{ createdBy: "config_ui" }`
+   - `agent_template_step` with default values
+   - `agent_template_version` with definition snapshot
+4. Updates conversation to link template
+5. Returns template info
+
+### Template Execution Flow
+
+**Trigger:** User clicks "Run" in Playground
+
+**Process:**
+1. **Validation**
+   - Validate template exists and is active
+   - Validate all input submissions exist and belong to organization
+
+2. **Run Initialization** (`createAgentRun`)
+   - Create `agent_run` record with status "pending"
+   - Update status to "running"
+
+3. **Input Merging** (if multiple submissions)
+   - If template has multiple input submissions for first step
+   - Create merged submission combining all inputs via `createMergedSubmission`
+   - Copies all groups and items from source submissions
+   - Uses "merged" as `source_type` for item versions
+
+4. **Step Iteration** (for each step in order)
+
+   a. **Output Submission Creation**
+      - Create output submission with type "transformation"
+      - Name format: `{template.name} • {step.name}`
+
+   b. **Run Step Creation** (`createAgentRunStep`)
+      - Create `agent_run_step` record
+      - Link input submission (from previous step if `dependsOn` set, else original)
+      - Link output submission
+      - Store step config in `input` JSONB
+      - Status starts as "pending"
+
+   c. **Step Execution** (via configurable `runStepExecutor`)
+      - Update step status to "running"
+      - Execute step logic (LLM call, transformations, etc.)
+      - Returns `StepExecutionResult` containing:
+        - `status`: "completed", "failed", or "skipped"
+        - `reasoning`: Optional LLM reasoning
+        - `error`: Optional error message
+        - `output.data`: Structured output data
+        - `toolCalls[]`: List of tool invocations
+        - `transformations[]`: List of data transformations
+        - `artifacts[]`: Generated artifacts (charts, tables, etc.)
+
+   d. **Tool Call Logging**
+      - For each tool call in result
+      - Create `agent_tool_call` record with args, result, latency
+
+   e. **Transformation Processing**
+      - Get selected input versions from input submission
+      - For each transformation:
+        - Create `agent_transformation` record
+        - Persist output data via `upsertTransformationOutputVersions`:
+          - Maps output data to form fields
+          - Creates submission groups if needed
+          - Creates submission items if needed
+          - Creates `submission_item_version` records with:
+            - `source_type`: "transformation"
+            - `source_id`: transformation ID
+            - `selected`: true
+          - Deselects previous versions for same items
+
+   f. **Artifact Creation**
+      - For each artifact in result
+      - Create `agent_artifact` record
+      - Optionally mirror to conversation memory (legacy format)
+
+   g. **Step Finalization**
+      - Update step status to final state (completed/failed/skipped)
+      - Store reasoning, error, output
+      - Record output submission ID for next step
+      - If failed, mark entire run as failed and stop
+
+   h. **Error Handling**
+      - Catch any exceptions during step execution
+      - Update step and run status to "failed"
+      - Store detailed error messages
+      - Stop execution (no subsequent steps run)
+
+5. **Run Completion**
+   - If all steps succeed, mark run as "completed"
+   - Otherwise, run remains in "failed" state
+   - Return run ID and final status
+
+### Data Transformation Flow
+
+**Purpose:** Map data from input submission to output submission structure
+
+**Process:**
+1. **Parse Output Data**
+   - Output data is a nested object: `{ [groupName]: groupData }`
+   - `groupData` can be:
+     - Single group: `{ [fieldName]: { value, refId? } }`
+     - Multi group: `[{ [fieldName]: { value, refId? } }, ...]`
+
+2. **Map to Form Structure**
+   - Load output form's groups and fields
+   - Match output data by `groupName:fieldName`
+   - Build map of required submission items:
+     - `(formGroupId, groupOrder, formFieldId)` → `{ value, refId? }`
+
+3. **Create/Update Submission Groups**
+   - For each unique `(formGroupId, groupOrder)`:
+     - Check if `submission_group` exists
+     - Create if missing
+
+4. **Create/Update Submission Items**
+   - For each unique `(submissionGroupId, formFieldId)`:
+     - Check if `submission_item` exists
+     - Create if missing
+
+5. **Create Item Versions**
+   - For each item value:
+     - Deselect previous versions for this item
+     - Create new `submission_item_version`:
+       - `source_type`: "transformation"
+       - `source_id`: transformation ID
+       - `value`: normalized value (string or number)
+       - `selected`: true
+     - Track `refId` if provided (links back to input data)
+
+6. **Return Results**
+   - List of created output version IDs
+   - List of referenced input version IDs
+
+## API Functions
+
+### Server Functions (apps/main/src/routes/_authed/_app/agents/-fns.ts)
+
+#### Template Management
+- `createAgentTemplateFn`: Create new template (single step)
+- `updateAgentTemplateFn`: Update template name/description
+- `updateAgentTemplateStepFn`: Update step configuration
+- `executeAgentTemplateFn`: Execute template against submissions
+
+#### Run Management
+- `getAgentConversationRunsFn`: Get runs for conversation
+- `getAgentRunTimelineFn`: Get detailed run timeline with steps, tools, transformations
+
+#### Configuration
+- `getAgentConfigDataFn`: Load config page data (tools, templates, forms, submissions)
+- `getAgentToolsFn`: Get tool configurations
+- `updateToolDescriptionFn`: Update custom tool description
+
+#### Conversation Management
+- `createAgentFn`: Create new agent conversation
+- `getAgentConversationsFn`: Get all agent conversations
+- `getAgentDataFn`: Get agent conversation with messages and context
+- `renameAgentFn`: Rename conversation
+- `deleteAgentFn`: Soft delete conversation
+
+#### Context Management
+- `getRelevantSubmissionsFn`: Get relevant submissions for context
+- `getRecentItemsFn`: Get recent deals/submissions/forms (for "Add to Context")
+- `addToContextFn`: Add item to conversation memory
+- `deleteContextFn`: Remove item from conversation memory
+- `deleteArtifactFn`: Remove artifact from conversation memory
+
+#### Document Management
+- `insertDocumentAgentFn`: Create document and extraction records
+- `uploadDocumentAgentFn`: Upload document and trigger extraction workflow
+
+#### Citations
+- `getCitations`: Get citations by IDs with extraction info
+- `extractCitationIds`: Extract citation IDs from messages (supports old `[citation:ID]` and new `<cite data-id="ID">` formats)
+
+### DAL Functions (libs/dal/src/agent-runtime.ts)
+
+#### Runtime Queries
+- `getRuntimeSubmission`: Get submission for runtime (with org validation)
+- `getConversationRuns`: Get runs for conversation (paginated)
+- `getRunTimeline`: Get full run with steps, tool calls, transformations, artifacts
+
+#### Run Management
+- `createAgentRun`: Create new run record
+- `updateAgentRunStatus`: Update run status (pending → running → completed/failed/cancelled)
+- `createAgentRunStep`: Create step record
+- `updateAgentRunStepStatus`: Update step status
+
+#### Template Queries
+- `getAgentTemplateWithSteps`: Get template with ordered steps
+
+#### Data Operations
+- `createTypedSubmission`: Create submission with specific type
+- `createMergedSubmission`: Create submission by merging multiple sources
+- `getOutputFieldTargets`: Get form field targets for output mapping
+- `getOutputFormGroups`: Get form groups with fields for output
+- `getSelectedSubmissionItemVersionIds`: Get currently selected version IDs
+
+#### Transformation & Logging
+- `createTransformation`: Create transformation record
+- `upsertTransformationOutputVersions`: Persist transformation outputs to submission
+- `logAgentToolCall`: Log tool call with args, result, latency
+- `createAgentArtifact`: Create artifact and optionally mirror to conversation memory
+
+### Core Execution (libs/core/src/agents/execute-template.ts)
+
+#### `createExecuteTemplate({ runStepExecutor? })`
+
+Factory function that creates a template executor with customizable step execution logic.
+
+**Parameters:**
+- `runStepExecutor`: Optional function to execute individual steps (defaults to skip-all executor)
+
+**Returns:** `executeTemplate` function
+
+#### `executeTemplate({ member, input })`
+
+Main orchestration function that coordinates the full template execution flow.
+
+**Parameters:**
+- `member`: Member performing the execution
+- `input`: `ExecuteTemplateInput` with:
+  - `conversationId`
+  - `templateId`
+  - `inputSubmissionId` (or `inputSubmissionIds` for multiple)
+  - `triggerType` (default: "manual")
+
+**Returns:** `ExecuteTemplateResult`
+- On success: `{ ok: true, code: 200, data: { runId, status } }`
+- On error: `{ ok: false, code: 404|500, error: "message" }`
+
+**Process:** (See "Template Execution Flow" above for details)
+
+#### `RunStepExecutor` Interface
+
+Custom step executors must implement this interface:
+
+```typescript
+type RunStepExecutor = (input: RunStepExecutorInput) => Promise<StepExecutionResult>;
+
+type RunStepExecutorInput = {
+  member: Member;
+  runId: string;
+  runStepId: string;
+  step: {
+    id: string;
+    stepKey: string;
+    stepOrder: number;
+    name: string;
+    description: string;
+    prompt: string;
+    outputKey: string;
+    toolsAllowed: string[];
+    parentStepId: string | null;  // Updated: parent step reference
+    outputFormId: string;
+  };
+  inputSubmissionId: string;  // NOTE: Derived from parent step or run input
+  outputSubmissionId: string;
+  outputFormGroups: FormGroupHydrated[];
+  tools?: Record<string, unknown>; // Filtered tools if toolsAllowed specified
+};
+
+type StepExecutionResult = {
+  status: "completed" | "failed" | "skipped";
+  reasoning?: string;
+  error?: string;
+  output?: { data: OutputData };
+  toolCalls?: ToolCallLog[];
+  transformations?: TransformationLog[];
+  artifacts?: ArtifactOutput[];
+};
+
+// NOTE: This interface shows the intended design but code needs updating to match new schema
+```
+
+## Incomplete Features & TODOs
+
+### High Priority
+
+1. **Multi-Step Template Creation**
+   - Current: Chat interface and form only create single-step templates
+   - Schema supports: Multiple steps with parent-child relationships via `parent_step_id`
+   - **TODO:** Build UI for creating templates with multiple steps
+   - Requires:
+     - Step sequence builder
+     - Parent step selection for dependencies
+     - Output → Input form compatibility validation
+   - Locations:
+     - Form creator: [apps/main/src/routes/_authed/_app/agents/-components/agent-template-creator.tsx](../apps/main/src/routes/_authed/_app/agents/-components/agent-template-creator.tsx)
+     - Server function: [apps/main/src/routes/_authed/_app/agents/-fns.ts:613-722](../apps/main/src/routes/_authed/_app/agents/-fns.ts#L613-722)
+
+2. **Tool Selection per Step**
+   - Schema supports: `tools_allowed` array to restrict tools per step (empty = all allowed)
+   - UI limitation: Only shows current tools in read-only mode
+   - **TODO:** Implement UI for selecting/deselecting allowed tools per step
+   - Locations:
+     - Schema: [libs/db/src/schemas/conversations.ts:130](../libs/db/src/schemas/conversations.ts#L130)
+     - UI: [apps/main/src/routes/_authed/_app/agents/$agentId/config.tsx:482-490](../apps/main/src/routes/_authed/_app/agents/$agentId/config.tsx#L482-490)
+
+3. **Template Versioning UI**
+   - Schema exists: `agent_template_version` table with `definition` snapshots
+   - Runtime support: Runs link to `template_version_id`
+   - UI missing: No version history, no version selection, no diff viewer
+   - **TODO:** Implement version management UI (lower priority)
+   - Intended behavior:
+     - Each template has an "active version" pointer
+     - Runs link to specific template version
+     - Allows template evolution without breaking historical runs
+   - Location: [libs/db/src/schemas/conversations.ts:147-165](../libs/db/src/schemas/conversations.ts#L147-165)
+
+### Medium Priority
+
+4. **Execution Layer Updates for New Schema**
+   - **Current:** Code still references old schema fields (`inputSubmissionId`, `inputSubmissionIds`, `dependsOn` array)
+   - **TODO:** Update execution layer to use new parent-child model
+   - Changes needed:
+     - Remove `inputSubmissionId`/`inputSubmissionIds` parameters
+     - Derive input from parent step's output submission
+     - Update step dependency resolution to use `parent_step_id`
+     - Fix `pickStepInputSubmissionId` logic
+   - Locations:
+     - Core execution: [libs/core/src/agents/execute-template.ts](../libs/core/src/agents/execute-template.ts)
+     - DAL functions: [libs/dal/src/agent-runtime.ts](../libs/dal/src/agent-runtime.ts)
+     - Type definitions: [libs/schemas/src/core/agent-runtime.ts](../libs/schemas/src/core/agent-runtime.ts)
+
+5. **Automated Template Triggers**
+   - Schema supports: `trigger_type` field on runs
+   - Currently only: "manual" triggers via playground
+   - **TODO:** Implement automated triggers (e.g., on submission creation, on deal update)
+   - Potential trigger types: "on_submission_created", "on_submission_updated", "scheduled"
+   - Location: [libs/db/src/schemas/conversations.ts:295](../libs/db/src/schemas/conversations.ts#L295)
+
+6. **Retry Policies**
+   - Schema exists: `retry_policy` JSONB on template steps
+   - Not implemented: No retry logic in execution layer
+   - **TODO:** Implement step retry on failure with configurable policies
+   - Would use `agent_run_step.attempt` field to track retry attempts
+   - Location: [libs/db/src/schemas/conversations.ts:131](../libs/db/src/schemas/conversations.ts#L131)
+
+### Low Priority / Nice-to-Have
+
+7. **Template Execution Metrics**
+   - Partial: Step `duration_ms` and tool `latency_ms` tracked
+   - **TODO:** Aggregate metrics, percentiles, success rates, cost tracking
+   - Potential: Dashboard view of template performance over time
+
+8. **Template Marketplace / Sharing**
+   - Schema supports: `is_system` flag for system-provided templates
+   - **TODO:** System templates library, import/export, template sharing
+
+9. **Visual Step Builder**
+   - Current: Text-based step configuration
+   - **TODO:** Drag-and-drop step builder with visual dependency graph
+   - Would visualize `parent_step_id` relationships as a flowchart
+
+### Known Issues
+
+10. **Model Configuration Cleanup**
+    - Location: [apps/main/src/routes/_authed/_app/agents/-fns.ts:239](../apps/main/src/routes/_authed/_app/agents/-fns.ts#L239)
+    - Comment: `// TODO: remove this nonsense once we merge all the models we use`
+    - Issue: Model configuration handling needs cleanup/consolidation
+
+11. **Schema-Code Mismatch**
+    - **Critical:** Execution code not yet updated for new schema
+    - Old code references removed fields: `inputFormId`, `inputFormIds`, `dependsOn`, `inputSubmissionId`, `inputSubmissionIds`
+    - New schema uses: template-level `input_form_id`, step `parent_step_id`, parent's output as input
+    - **Impact:** Template execution will fail until code is updated
+    - **Action Required:** Update all execution layer code to match new schema
+
+## Key Design Patterns
+
+### 1. DalResult Pattern
+All DAL functions return `DalResult<T>`:
+```typescript
+type DalResult<T> =
+  | { ok: true; data: T }
+  | { ok: false; error: string }
+```
+Enables consistent error handling and type-safe unwrapping.
+
+### 2. Separation of Concerns
+- **UI Layer** (apps/main): React components, routing, user interaction
+- **API Layer** (apps/main/-fns.ts): Server functions with auth middleware
+- **DAL Layer** (libs/dal): Database queries with organization-scoped access
+- **Core Layer** (libs/core): Business logic, template execution orchestration
+- **Schema Layer** (libs/schemas): Type definitions and validation
+
+### 3. Template Execution Flexibility
+The `createExecuteTemplate` factory allows customizing step execution logic while reusing orchestration:
+- Core handles: run creation, step iteration, error handling, status updates
+- Custom executor handles: actual AI/LLM logic, tool calling, data transformation
+
+### 4. Optimistic UI Updates
+Config page uses local state for edits, only persisting on explicit save:
+- Reduces API calls
+- Provides instant feedback
+- Shows "modified" state clearly
+- Prevents accidental changes
+
+### 5. Multi-Input Support
+System supports multiple input submissions:
+- First step merges inputs if multiple provided
+- Subsequent steps use previous step's output (via `dependsOn`)
+- Maintains provenance via `refId` in output values
+
+## Testing & Debugging
+
+### Playground Testing Flow
+1. Navigate to `/agents/{id}/config` → Playground tab
+2. Select template from dropdown
+3. Select input submissions (grouped by required forms)
+4. Click "Run" → observe run creation
+5. Check "Recent Runs" panel for new run
+6. Click run → observe timeline with steps
+7. Inspect step details: inputs, outputs, tool calls, errors
+8. Click submission links to verify data transformations
+
+### Debugging Failed Runs
+1. Check run status in "Recent Runs" (red badge for failed)
+2. Load timeline to see which step failed
+3. Review step's `error` field for details
+4. Check `reasoning` for LLM's explanation
+5. Inspect tool calls for failed tool invocations
+6. Verify input submission data structure matches expected form
+7. Check output form field mappings in `upsertTransformationOutputVersions`
+
+### Database Inspection
+```sql
+-- Get run with all steps
+SELECT r.id, r.status, r.created_at,
+       s.step_order, s.name, s.status, s.error
+FROM agent_run r
+LEFT JOIN agent_run_step s ON s.run_id = r.id
+WHERE r.id = 'run-id'
+ORDER BY s.step_order;
+
+-- Get tool calls for a step
+SELECT tool_name, status, latency_ms, called_at
+FROM agent_tool_call
+WHERE run_step_id = 'step-id'
+ORDER BY called_at;
+
+-- Get transformations for a step
+SELECT kind, expression, metadata
+FROM agent_transformation
+WHERE run_step_id = 'step-id';
+
+-- Get output versions created by transformation
+SELECT siv.id, siv.value, siv.selected,
+       ff.name as field_name
+FROM submission_item_version siv
+JOIN submission_item si ON si.id = siv.submission_item_id
+JOIN form_field ff ON ff.id = si.form_field_id
+WHERE siv.source_type = 'transformation'
+  AND siv.source_id = 'transformation-id';
+```
+
+## Future Enhancements
+
+### Short-term
+- ✅ **DONE:** Fix `depends_on` schema (array → single `parent_step_id` FK)
+- ✅ **DONE:** Move `input_form_id` from step to template level
+- Update execution layer code to match new schema
+- Implement tool selection UI
+- Add multi-step template creation UI
+- Improve error messages with suggested fixes
+
+### Medium-term
+- Template versioning UI with diff viewer
+- Automated triggers on submission/deal events
+- Retry policies with backoff strategies
+- Multi-step template creation flow
+- Visual step dependency graph
+
+### Long-term
+- Template marketplace with system templates
+- Performance analytics dashboard
+- Cost tracking per run/step
+- Template testing framework
+- Visual step builder with drag-and-drop
+- Template composition (templates as steps)
+
+## Related Documentation
+- [CLAUDE.md](../CLAUDE.md) - Development guidelines and monorepo structure
+- Database schema definitions: [libs/db/src/schemas/conversations.ts](../libs/db/src/schemas/conversations.ts)
+- DAL implementation: [libs/dal/src/agent-runtime.ts](../libs/dal/src/agent-runtime.ts)
+- Core execution: [libs/core/src/agents/execute-template.ts](../libs/core/src/agents/execute-template.ts)
+- Schema types: [libs/schemas/src/core/agent-runtime.ts](../libs/schemas/src/core/agent-runtime.ts)