sea-dev 1.0.0

package/docs/impl-p0-form-builder-improvements.md

@@ -0,0 +1,683 @@
+# Implementation Plan: P0 Form Builder Improvements
+
+This document provides detailed implementation plans for all P0 items from the Form Builder Strategy PRD.
+
+---
+
+## Overview
+
+| Item | Summary | Files to Change | Effort |
+|------|---------|-----------------|--------|
+| **A** | Enable derived fields in chat builder | `tools.ts` | ~30 min |
+| **B** | Enhanced system prompt with form knowledge | `prompts.ts` | ~45 min |
+| **G** | Extraction guidance in prompts | `prompts.ts` | ~15 min |
+| **K** | Submission-aware edit restrictions | `tools.ts`, `prompts.ts`, `form.ts` | ~1 hr |
+
+**Total estimated implementation time:** ~2.5 hours
+
+---
+
+## P0-A: Enable Derived Fields in Chat Builder
+
+### Problem
+The chat builder schema (`formGroupSchema` in `tools.ts`) omits derived fields. Users cannot create assessment/AI fields through the chat interface.
+
+### Current State
+
+**File:** `apps/main/src/llm/tools.ts:42-57`
+
+```typescript
+const formGroupSchema = z.object({
+  id: z.string().optional().describe("..."),
+  name: z.string().describe("The user-facing name of this group"),
+  hint: z.string().describe("LLM-facing hint for the group"),
+  multi: z.boolean().describe("Whether this group should allow multiple entries..."),
+  fields: z.array(formFieldSchema),
+  // NOTE: No derived field array!
+});
+```
+
+**File:** `libs/dal/src/form.ts:30-50` - `getForm()` does NOT include derived fields:
+
+```typescript
+with: {
+  groups: {
+    with: {
+      fields: { orderBy: asc(schema.formField.order) },
+      // NOTE: Missing `derived` relation!
+    },
+  },
+},
+```
+
+### Solution
+
+#### Step 1: Add derived schema to tools.ts
+
+**File:** `apps/main/src/llm/tools.ts`
+
+Add after line 40 (after `formFieldSchema`):
+
+```typescript
+const formDerivedSchema = z.object({
+  id: z
+    .string()
+    .optional()
+    .describe(
+      "Provide this only for existing derived fields that have an id. Don't generate an id for new items",
+    ),
+  name: z.string().describe("The user-facing name of this derived/assessment field"),
+  prompt: z.string().describe(
+    "The LLM prompt that will be used to generate this field's value after extraction. " +
+    "Example: 'Based on the extracted financial data, assess whether this borrower shows signs of financial distress. Return YES or NO with a brief explanation.'"
+  ),
+});
+```
+
+Update `formGroupSchema` (line 42-57):
+
+```typescript
+const formGroupSchema = z.object({
+  id: z
+    .string()
+    .optional()
+    .describe(
+      "Provide this only for existing items that have an id. Don't generate an id for new groups",
+    ),
+  name: z.string().describe("The user-facing name of this group"),
+  hint: z.string().describe("LLM-facing hint for the group"),
+  multi: z
+    .boolean()
+    .describe(
+      "Whether this group should allow multiple entries, eg data this is repeated for multiple years",
+    ),
+  fields: z.array(formFieldSchema),
+  derived: z
+    .array(formDerivedSchema)
+    .optional()
+    .describe(
+      "Assessment fields that are populated by AI AFTER extraction completes. " +
+      "Use these for judgments, risk flags, or calculations based on extracted data. " +
+      "Do NOT use for direct extraction from documents."
+    ),
+});
+```
+
+#### Step 2: Update getForm() to include derived fields
+
+**File:** `libs/dal/src/form.ts:30-50`
+
+```typescript
+export async function getForm({ member, formId }: { member: Member; formId: string }) {
+  const form = await db.query.form.findFirst({
+    where: and(
+      eq(schema.form.organizationId, member.organizationId),
+      isNull(schema.form.deletedAt),
+      eq(schema.form.id, formId),
+    ),
+    with: {
+      groups: {
+        with: {
+          fields: {
+            orderBy: asc(schema.formField.order),
+          },
+          derived: true,  // ADD THIS LINE
+        },
+        orderBy: asc(schema.formGroup.order),
+      },
+    },
+  });
+  invariant(form, `no form found for ${formId}`);
+  return form;
+}
+```
+
+#### Step 3: Update persistFormInner() to handle derived fields
+
+**File:** `apps/main/src/llm/tools.ts:107-189`
+
+Add derived field handling in the `persistFormInner` function. After processing fields for each group:
+
+```typescript
+// Inside the group processing loop, after field handling (around line 187):
+
+// Handle derived fields
+if (group.derived && group.derived.length > 0) {
+  const formGroupId = group.id || newGroup.id;
+
+  // Get existing derived fields for this group
+  const existingDerived = await db.query.formDerived.findMany({
+    where: eq(schema.formDerived.formGroupId, formGroupId),
+  });
+
+  // Delete derived fields that are no longer in the new list
+  const derivedIdsToKeep = group.derived.filter(d => d.id).map(d => d.id);
+  const derivedIdsToDelete = existingDerived
+    .filter(ed => !derivedIdsToKeep.includes(ed.id))
+    .map(d => d.id);
+
+  if (derivedIdsToDelete.length > 0) {
+    await db.delete(schema.formDerived).where(inArray(schema.formDerived.id, derivedIdsToDelete));
+  }
+
+  // Upsert derived fields
+  for (const derived of group.derived) {
+    if (derived.id) {
+      await db
+        .update(schema.formDerived)
+        .set({ name: derived.name, prompt: derived.prompt })
+        .where(eq(schema.formDerived.id, derived.id));
+    } else {
+      await db.insert(schema.formDerived).values({
+        formGroupId,
+        name: derived.name,
+        prompt: derived.prompt,
+      });
+    }
+  }
+}
+```
+
+#### Step 4: Update type for Group
+
+**File:** `apps/main/src/llm/tools.ts:60`
+
+```typescript
+type Group = z.infer<typeof formGroupSchema>;
+type DerivedField = z.infer<typeof formDerivedSchema>;
+```
+
+### Testing Checklist
+- [ ] Chat can fetch forms with existing derived fields
+- [ ] Chat can add new derived fields to a group
+- [ ] Chat can update existing derived field names and prompts
+- [ ] Chat can remove derived fields (with confirmation)
+- [ ] Forms without derived fields still work correctly
+
+---
+
+## P0-B: Enhanced System Prompt with Form Knowledge
+
+### Problem
+The form system prompt (in `prompts.ts`) only knows form name and workflow name. It doesn't understand form primitives, best practices, or semantic meaning of form components.
+
+### Current State
+
+**File:** `apps/main/src/llm/prompts.ts:37-99`
+
+```typescript
+async function formSystemPrompt({ member, conversationId }) {
+  // Only fetches form.name and workflow.name
+  // Generic instructions about fetch/draft/persist phases
+  // No knowledge of:
+  // - Collection groups (multi)
+  // - Derived fields
+  // - Field types and when to use them
+  // - Extraction best practices
+}
+```
+
+### Solution
+
+**File:** `apps/main/src/llm/prompts.ts`
+
+Replace the `formSystemPrompt` function:
+
+```typescript
+async function formSystemPrompt({
+  member,
+  conversationId,
+}: {
+  member: MemberHydrated;
+  conversationId: string;
+}) {
+  const form = await db.query.form.findFirst({
+    where: and(
+      eq(schema.form.organizationId, member.organizationId),
+      eq(schema.form.conversationId, conversationId),
+    ),
+    columns: { name: true },
+    with: { workflow: { columns: { name: true } } },
+  });
+  invariant(form);
+
+  const context = `
+- You are an assistant developed by sea.dev to help users with their data processing needs.
+- The current date is ${formatISODate(new Date())}
+- The current user is ${member.user.name}, who is a member of ${member.organization.name}
+- The current form is called "${form.name}"
+- It is part of a workflow called "${form.workflow.name}"
+  `;
+
+  const formKnowledge = `
+## Form Structure Knowledge
+
+You are building forms that serve as **data extraction targets**. Documents (PDFs, bank statements, financial reports) are uploaded and AI extracts structured data into the shape defined by the form.
+
+### Form Primitives
+
+1. **Groups** - Logical sections that organize related fields
+   - Regular groups: For single-instance data (e.g., "Company Information")
+   - Collection groups (\`multi: true\`): For repeating data (e.g., "Transactions", "Monthly Figures")
+
+2. **Fields** - Individual data points with types:
+   - \`string\`: Short text (names, IDs, descriptions)
+   - \`string-long\`: Long text (notes, addresses)
+   - \`number\`: Numeric values (quantities, counts)
+   - \`money\`: Currency amounts (always use for financial figures)
+   - \`date\`: Dates (statement date, transaction date)
+   - \`email\`: Email addresses
+   - \`url\`: Web URLs
+   - \`boolean\`: Yes/No values
+
+3. **Derived Fields** - AI assessment fields that run AFTER extraction
+   - Use for: Risk flags, quality assessments, calculations, judgments
+   - NOT for: Direct document extraction
+   - Example: "Based on the transactions, is this account showing signs of financial distress?"
+
+### Collection Groups Best Practices
+
+When creating collection groups (multi: true):
+- **Pin the key identifier first** - The first field should be what other fields relate to
+  - For transactions: Date first
+  - For periods: Period/Year first
+  - For line items: Item name first
+- Common patterns:
+  - Bank statement transactions: Date, Description, Amount, Balance
+  - Income statement periods: Period, Revenue, Expenses, Net Income
+  - Balance sheet items: Item, Value, Notes
+
+### Field Hints Are Critical
+
+Field hints tell the extraction AI where to find data. Be specific:
+- ✅ Good: "Net income after tax, typically on line 28 of the income statement"
+- ❌ Bad: "Net income"
+- ✅ Good: "Transaction date in DD/MM/YYYY format from the first column"
+- ❌ Bad: "Date"
+
+### Extractability Constraint
+
+**Only include fields that could conceivably be extracted from an uploaded document.**
+
+Cannot be extracted (avoid these):
+- Data from external APIs (credit scores, company registry lookups)
+- Calculated aggregates across multiple documents
+- Real-time data that changes after document creation
+- Information requiring human judgment not in the document
+
+Can be extracted:
+- Any text, number, or date visible in the document
+- Calculated values shown in the document (totals, subtotals)
+- Information that can be inferred from document content
+`;
+
+  const text = `
+You are a helpful AI agent that builds and updates data extraction forms.
+
+${formKnowledge}
+
+## Workflow
+
+If the user asks you to update the form, follow these steps:
+
+**Phase 1: Retrieve**
+You MUST always begin by calling \`fetchCurrentForm()\` to get the latest form structure.
+
+**Phase 2: Draft**
+1. Analyze the user's request against the current form
+2. If the request is ambiguous, ask clarifying questions
+3. Draft changes following the form structure knowledge above
+4. Write a concise summary of proposed changes
+
+**Phase 3: Confirm & Persist**
+1. Present your summary and ask for explicit confirmation
+2. If confirmed, call \`persistForm()\` with the complete updated form
+3. Confirm what was done
+
+## General Rules
+- Always fetch the current form first - never assume its contents
+- Any deletion or major restructuring is [DESTRUCTIVE] - require confirmation
+- When adding collection groups, always consider what the key/pinned first field should be
+- Prefer \`money\` type for any financial amounts
+- Write detailed field hints that help extraction
+
+Background context:
+${context}
+  `;
+
+  return text;
+}
+```
+
+### Testing Checklist
+- [ ] Chat understands and can explain collection groups
+- [ ] Chat suggests appropriate first fields for collections
+- [ ] Chat uses correct field types (money for amounts)
+- [ ] Chat writes detailed field hints
+- [ ] Chat warns about non-extractable fields
+
+---
+
+## P0-G: Extraction Guidance in Prompts
+
+### Problem
+Users don't know what makes a good form for AI extraction. The chat should proactively guide form design.
+
+### Solution
+
+This is largely addressed by P0-B above. The form knowledge section includes extraction guidance.
+
+**Additional enhancement** - Add specific examples to the prompt:
+
+Add to the `formKnowledge` section in `prompts.ts`:
+
+```typescript
+const extractionExamples = `
+### Common Form Patterns
+
+**Bank Statement Form:**
+\`\`\`
+Group: Account Information (multi: false)
+  - Account Number (string): "Bank account number, typically at top of statement"
+  - Account Holder (string): "Name of the account holder"
+  - Statement Date (date): "Statement period end date"
+  - Opening Balance (money): "Balance at start of period"
+  - Closing Balance (money): "Balance at end of period"
+
+Group: Transactions (multi: true)
+  - Date (date): "Transaction date in first column" [PINNED FIRST]
+  - Description (string): "Transaction description or reference"
+  - Amount (money): "Transaction amount, negative for debits"
+  - Running Balance (money): "Balance after transaction"
+\`\`\`
+
+**Income Statement Form:**
+\`\`\`
+Group: Company Info (multi: false)
+  - Company Name (string): "Company name from header"
+  - Report Period (string): "Reporting period, e.g., 'FY 2024' or 'Q3 2024'"
+
+Group: Financial Lines (multi: true)
+  - Line Item (string): "Name of the financial line item" [PINNED FIRST]
+  - Amount (money): "Value for this line item"
+  - Prior Period (money): "Comparison value from prior period, if shown"
+
+Group: Assessment (multi: false)
+  - Revenue Trend (derived): "Based on revenue figures, is revenue growing, stable, or declining?"
+  - Profitability Flag (derived): "Is the company profitable based on net income? YES/NO"
+\`\`\`
+`;
+```
+
+### Testing Checklist
+- [ ] Chat can suggest bank statement form structure
+- [ ] Chat can suggest income statement form structure
+- [ ] Examples help users understand the pattern
+
+---
+
+## P0-K: Submission-Aware Edit Restrictions
+
+### Problem
+When a form has existing submissions, deleting fields/groups fails silently due to foreign key constraints. The chat doesn't know about submissions and can't warn users.
+
+### Current Behavior
+1. User asks: "Remove the Revenue field"
+2. Chat attempts `persistForm()` with field removed
+3. Database rejects (FK constraint on `submission_item.form_field_id`)
+4. User sees generic error
+
+### Solution
+
+#### Step 1: Add new tool to check form editability
+
+**File:** `apps/main/src/llm/tools.ts`
+
+Add new tool after `fetchCurrentForm`:
+
+```typescript
+export function fetchFormEditability({ member, formId }: { member: Member; formId: string }) {
+  return tool({
+    description:
+      "Check what edits are allowed on this form. MUST be called before any form modifications " +
+      "to understand if the form has submissions that restrict certain changes.",
+    inputSchema: z.object({}),
+    execute: async () => {
+      try {
+        // Count submissions for this form
+        const [result] = await db
+          .select({ count: sql<number>`count(*)::int` })
+          .from(schema.submission)
+          .where(
+            and(
+              eq(schema.submission.formId, formId),
+              eq(schema.submission.organizationId, member.organizationId),
+              isNull(schema.submission.deletedAt),
+            ),
+          );
+
+        const submissionCount = result?.count ?? 0;
+        const hasSubmissions = submissionCount > 0;
+
+        return {
+          ok: true,
+          data: {
+            hasSubmissions,
+            submissionCount,
+            restrictions: {
+              canDeleteFields: !hasSubmissions,
+              canDeleteGroups: !hasSubmissions,
+              canChangeFieldTypes: !hasSubmissions,
+              canRenameFields: true,
+              canUpdateHints: true,
+              canAddFields: true,
+              canAddGroups: true,
+              canAddDerivedFields: true,
+              canReorderFields: true,
+            },
+            message: hasSubmissions
+              ? `This form has ${submissionCount} submission(s) with data. ` +
+                `You CAN: add fields, rename fields, update hints, add derived fields, reorder fields. ` +
+                `You CANNOT: delete fields, delete groups, or change field types (would break existing data).`
+              : "This form has no submissions. All edits are allowed.",
+          },
+        };
+      } catch (err) {
+        return returnErr(err);
+      }
+    },
+  });
+}
+```
+
+#### Step 2: Update system prompt to require editability check
+
+**File:** `apps/main/src/llm/prompts.ts`
+
+Update the workflow section:
+
+```typescript
+const text = `
+You are a helpful AI agent that builds and updates data extraction forms.
+
+${formKnowledge}
+
+## Workflow
+
+If the user asks you to update the form, follow these steps:
+
+**Phase 1: Retrieve & Check Restrictions**
+1. Call \`fetchCurrentForm()\` to get the latest form structure
+2. Call \`fetchFormEditability()\` to check if the form has submissions
+
+**Phase 2: Draft**
+1. Review the editability restrictions from Phase 1
+2. If the user requests a restricted action (delete field/group, change type) AND the form has submissions:
+   - Explain why it's not allowed
+   - Suggest alternatives:
+     - Add a new field instead of modifying existing
+     - Update field hints to improve extraction
+     - Clone the form if they need a fresh start
+3. For allowed changes, draft the updates following form structure best practices
+4. Write a concise summary of proposed changes
+
+**Phase 3: Confirm & Persist**
+1. Present your summary and ask for explicit confirmation
+2. If confirmed, call \`persistForm()\` with the complete updated form
+3. Confirm what was done
+
+## Edit Restrictions
+
+Forms with existing submissions have data integrity constraints:
+
+| Action | Allowed? | Why |
+|--------|----------|-----|
+| Add new fields | ✅ Yes | Existing data unaffected |
+| Add new groups | ✅ Yes | Existing data unaffected |
+| Add derived fields | ✅ Yes | These are computed, not stored |
+| Rename fields | ✅ Yes | Only label changes, data preserved |
+| Update hints | ✅ Yes | Only affects future extraction |
+| Reorder fields | ✅ Yes | Display only, data preserved |
+| Delete fields | ❌ No | Would orphan submission data |
+| Delete groups | ❌ No | Would orphan submission data |
+| Change field type | ❌ No | Could corrupt existing values |
+
+If a user wants to make restricted changes, suggest:
+1. "Clone this form to create a fresh version you can fully edit"
+2. "Delete existing submissions first (if they're test data)"
+3. "Add a new field alongside the existing one"
+
+## General Rules
+- Always check editability before proposing changes
+- Any deletion or major restructuring is [DESTRUCTIVE] - require confirmation
+- When a form has submissions, be extra careful about data integrity
+
+Background context:
+${context}
+  `;
+```
+
+#### Step 3: Register the new tool
+
+**File:** Where form chat tools are registered (likely in a route handler)
+
+Ensure `fetchFormEditability` is included in the tools passed to the chat:
+
+```typescript
+// In the form chat handler
+const tools = {
+  fetchCurrentForm: fetchCurrentForm({ member, formId }),
+  fetchFormEditability: fetchFormEditability({ member, formId }),  // ADD THIS
+  persistForm: persistForm({ member, formId }),
+};
+```
+
+#### Step 4: Add validation in persistFormInner (defense in depth)
+
+**File:** `apps/main/src/llm/tools.ts:107-189`
+
+Add validation at the start of `persistFormInner`:
+
+```typescript
+async function persistFormInner({
+  member,
+  formId,
+  groups,
+}: {
+  member: Member;
+  formId: string;
+  groups: Group[];
+}) {
+  const form = await db.query.form.findFirst({
+    where: and(eq(schema.form.organizationId, member.organizationId), eq(schema.form.id, formId)),
+    with: { groups: { with: { fields: true } } },
+  });
+  invariant(form);
+
+  // Check if form has submissions
+  const [submissionResult] = await db
+    .select({ count: sql<number>`count(*)::int` })
+    .from(schema.submission)
+    .where(
+      and(
+        eq(schema.submission.formId, formId),
+        eq(schema.submission.organizationId, member.organizationId),
+        isNull(schema.submission.deletedAt),
+      ),
+    );
+  const hasSubmissions = (submissionResult?.count ?? 0) > 0;
+
+  if (hasSubmissions) {
+    // Validate no destructive changes
+    const existingFieldIds = new Set(form.groups.flatMap(g => g.fields.map(f => f.id)));
+    const existingGroupIds = new Set(form.groups.map(g => g.id));
+
+    const newFieldIds = new Set(groups.flatMap(g => g.fields.filter(f => f.id).map(f => f.id)));
+    const newGroupIds = new Set(groups.filter(g => g.id).map(g => g.id));
+
+    // Check for deleted fields
+    for (const existingId of existingFieldIds) {
+      if (!newFieldIds.has(existingId)) {
+        throw new Error(
+          `Cannot delete field (ID: ${existingId}) because this form has existing submissions. ` +
+          `To delete fields, first remove all submissions or clone the form.`
+        );
+      }
+    }
+
+    // Check for deleted groups
+    for (const existingId of existingGroupIds) {
+      if (!newGroupIds.has(existingId)) {
+        throw new Error(
+          `Cannot delete group (ID: ${existingId}) because this form has existing submissions. ` +
+          `To delete groups, first remove all submissions or clone the form.`
+        );
+      }
+    }
+  }
+
+  // ... rest of existing logic
+}
+```
+
+### Testing Checklist
+- [ ] `fetchFormEditability` returns correct submission count
+- [ ] Chat calls `fetchFormEditability` before proposing changes
+- [ ] Chat explains restrictions when form has submissions
+- [ ] Chat suggests alternatives (clone, add new field)
+- [ ] `persistFormInner` throws clear error if restricted change attempted
+- [ ] Forms without submissions allow all operations
+
+---
+
+## Implementation Order
+
+Recommended order for implementation:
+
+1. **P0-K** (Submission-aware restrictions) - Most impactful UX fix
+2. **P0-A** (Derived fields) - Enables new functionality
+3. **P0-B + P0-G** (Enhanced prompts) - Can be done together
+
+---
+
+## File Change Summary
+
+| File | Changes |
+|------|---------|
+| `apps/main/src/llm/tools.ts` | Add `formDerivedSchema`, update `formGroupSchema`, add `fetchFormEditability`, update `persistFormInner` |
+| `apps/main/src/llm/prompts.ts` | Complete rewrite of `formSystemPrompt` with form knowledge |
+| `libs/dal/src/form.ts` | Update `getForm` to include derived fields |
+| Route handler | Register `fetchFormEditability` tool |
+
+---
+
+## Rollback Plan
+
+All changes are additive and backward-compatible:
+
+1. **Derived fields**: Existing forms without derived fields continue to work
+2. **Enhanced prompts**: Only changes AI behavior, no data changes
+3. **Editability check**: New tool, existing tools unchanged
+4. **Validation in persistFormInner**: Only prevents operations that would fail anyway
+
+If issues arise, changes can be reverted file-by-file without data migration.