@claritylabs/cl-sdk 0.2.0 → 0.3.1

package/README.md

@@ -1,6 +1,8 @@
-[Clarity Labs](https://claritylabs.inc) is an applied AI research lab building the infrastructure for AI to work safely with insurance.
+# CL-0 SDK
 
-AI agents are already executing complex tasks autonomously across industries, but insurance requires context, safeguards, and systems that don't exist yet. CL-0 SDK fills that gap: a shared intelligence layer that any product or agent can import to understand, reason about, and act on insurance.
+[Clarity Labs](https://claritylabs.inc) allows insurers to understand their clients as well as they know themselves. Having a better understanding of clients means insurers can automate servicing to reduce costs and identify coverage gaps to cross-sell products.
+
+CL-0 SDK is the open infrastructure layer that makes this possible: a shared intelligence system that any product or agent can import to understand, reason about, and act on insurance documents and workflows.
 
 ## Installation
 
@@ -10,284 +12,216 @@ npm install @claritylabs/cl-sdk
 
 ### Peer Dependencies
 
-CL-0 SDK requires the [Vercel AI SDK](https://sdk.vercel.ai) and pdf-lib:
-
 ```bash
-npm install ai pdf-lib
+npm install pdf-lib zod
 ```
 
-Then install a provider package for your model of choice:
-
+Optional (for SQLite storage):
 ```bash
-# Anthropic
-npm install @ai-sdk/anthropic
-
-# OpenAI
-npm install @ai-sdk/openai
-
-# Google
-npm install @ai-sdk/google
+npm install better-sqlite3
 ```
 
 ## Quick Start
 
-### Uniform Model (same model for all passes)
+### Document Extraction
+
+The v6 extraction pipeline uses a coordinator/worker pattern with provider-agnostic callbacks:
 
 ```typescript
-import { createAnthropic } from "@ai-sdk/anthropic";
-import { extractFromPdf, createUniformModelConfig, applyExtracted } from "@claritylabs/cl-sdk";
+import { createExtractor } from "@claritylabs/cl-sdk";
+import { anthropic } from "@ai-sdk/anthropic"; // or any provider
+import { generateText, generateObject } from "ai";
+
+const extract = createExtractor({
+  generateText: async ({ prompt, system, maxTokens }) => {
+    const { text, usage } = await generateText({
+      model: anthropic("claude-sonnet-4-6"),
+      prompt,
+      system,
+      maxTokens,
+    });
+    return { text, usage };
+  },
+  generateObject: async ({ prompt, system, schema, maxTokens }) => {
+    const { object, usage } = await generateObject({
+      model: anthropic("claude-sonnet-4-6"),
+      prompt,
+      system,
+      schema,
+      maxTokens,
+    });
+    return { object, usage };
+  },
+});
 
-const anthropic = createAnthropic();
 const pdfBase64 = "..."; // base64-encoded PDF
-
-const { extracted } = await extractFromPdf(pdfBase64, {
-  models: createUniformModelConfig(anthropic("claude-sonnet-4-6")),
-});
-const fields = applyExtracted(extracted);
+const result = await extract.extract(pdfBase64);
+console.log(result.document); // Structured InsuranceDocument
 ```
 
-### Any Provider
+### With PDF to Image Conversion
 
-CL-0 SDK is provider-agnostic — PDFs are sent as native files by default, which most providers support (Anthropic, Google, OpenAI, Mistral, Bedrock, Azure):
+For providers that don't support native PDF input:
 
 ```typescript
-import { createOpenAI } from "@ai-sdk/openai";
-import { extractFromPdf, createUniformModelConfig } from "@claritylabs/cl-sdk";
-
-const openai = createOpenAI();
-const { extracted } = await extractFromPdf(pdfBase64, {
-  models: createUniformModelConfig(openai("gpt-4o")),
-});
-```
-
-If your model doesn't support native PDF input, set `pdfContentFormat: "image"` and provide a `convertPdfToImages` callback. The SDK does not bundle a converter — use whatever library works in your runtime (pdf2pic, mupdf, pdfjs-dist, etc.):
-
-```typescript
-const { extracted } = await extractFromPdf(pdfBase64, {
-  models: createUniformModelConfig(yourModel),
-  pdfContentFormat: "image",
+const extract = createExtractor({
+  generateText: /* ... */,
+  generateObject: /* ... */,
   convertPdfToImages: async (pdfBase64, startPage, endPage) => {
-    // Return one { imageBase64, mimeType } per page
-    return pages;
+    // Convert PDF pages to images using your preferred library
+    return [
+      { imageBase64: "...", mimeType: "image/png" },
+      // ... one per page
+    ];
   },
 });
 ```
 
-### Fine-Grained Model Config
-
-Assign different models per pipeline role — use a fast model for classification/sections and a capable model for metadata/fallback:
+### Storage (Optional)
 
 ```typescript
-import { createAnthropic } from "@ai-sdk/anthropic";
-import { extractFromPdf, type ModelConfig } from "@claritylabs/cl-sdk";
-
-const anthropic = createAnthropic();
-const models: ModelConfig = {
-  classification: anthropic("claude-haiku-4-5-20251001"),  // fast, cheap
-  metadata: anthropic("claude-sonnet-4-6"),                // capable
-  sections: anthropic("claude-haiku-4-5-20251001"),        // fast, cheap
-  sectionsFallback: anthropic("claude-sonnet-4-6"),        // capable (fallback)
-  enrichment: anthropic("claude-haiku-4-5-20251001"),      // fast, cheap
-};
-
-const { extracted } = await extractFromPdf(pdfBase64, { models });
-```
+import { createExtractor } from "@claritylabs/cl-sdk";
+import { SQLiteDocumentStore, SQLiteMemoryStore } from "@claritylabs/cl-sdk/storage/sqlite";
 
-### Mixed Providers
+const documentStore = new SQLiteDocumentStore("./docs.db");
+const memoryStore = new SQLiteMemoryStore("./memory.db");
 
-```typescript
-import { createAnthropic } from "@ai-sdk/anthropic";
-import { createOpenAI } from "@ai-sdk/openai";
-import { extractFromPdf, type ModelConfig } from "@claritylabs/cl-sdk";
-
-const anthropic = createAnthropic();
-const openai = createOpenAI();
-
-const models: ModelConfig = {
-  classification: openai("gpt-4o-mini"),
-  metadata: anthropic("claude-sonnet-4-6"),
-  sections: openai("gpt-4o-mini"),
-  sectionsFallback: anthropic("claude-sonnet-4-6"),
-  enrichment: openai("gpt-4o-mini"),
-};
-
-const { extracted } = await extractFromPdf(pdfBase64, { models });
+const extract = createExtractor({
+  generateText: /* ... */,
+  generateObject: /* ... */,
+  documentStore,
+  memoryStore,
+});
 ```
 
-## What's Inside
-
-### Document Extraction Pipeline
+## Architecture
 
-A multi-pass system that turns insurance PDFs into structured, queryable data:
+### Provider-Agnostic Design
 
-- **Pass 0 — Classification**: Determines whether a document is a policy or a quote. Returns document type, confidence score, and supporting signals.
-- **Pass 1 — Metadata Extraction**: Extracts high-level metadata — carrier, policy/quote number, dates, premium, insured name, coverage table with limits and deductibles. Includes an early persistence callback (`onMetadata`) so metadata is saved immediately, surviving downstream failures.
-- **Pass 2 — Section Extraction**: Splits the document into page chunks (starting at 15 pages) and extracts structured sections in parallel (concurrency-limited, default 2). All model calls automatically retry on rate-limit errors with exponential backoff. Adaptive fallback: if a chunk's output is truncated (JSON parse failure), it re-splits into smaller chunks (10, then 5 pages), and escalates to the fallback model. Results are merged across chunks.
-- **Pass 3 — Enrichment**: A non-fatal pass that parses raw text into structured supplementary fields — regulatory context, complaint contacts, costs and fees, claims contacts.
+CL-0 SDK has **zero framework dependencies**. You provide simple callback functions:
 
-For quotes specifically, the pipeline also extracts premium breakdowns, subjectivities (conditions that must be met before binding), and underwriting conditions.
-
-### Application Processing
-
-End-to-end insurance application handling — from blank PDF to filled form:
-
-- **Application detection** — classifies whether a PDF is an insurance application form
-- **Field extraction** — reads every fillable field as structured data (text, numeric, currency, date, yes/no, table, and declaration fields)
-- **Auto-fill** — matches extracted fields against known business context to pre-populate answers
-- **Question batching** — organizes unfilled fields into topic-based batches for emailing the insured
-- **Answer parsing** — parses free-text replies back into structured field values
-- **PDF filling** — maps answers back onto the original PDF, supporting both AcroForm and flat PDF overlay
+```typescript
+type GenerateText = (params: {
+  prompt: string;
+  system?: string;
+  maxTokens: number;
+}) => Promise<{ text: string; usage?: TokenUsage }>;
+
+type GenerateObject<T> = (params: {
+  prompt: string;
+  system?: string;
+  schema: ZodSchema<T>;
+  maxTokens: number;
+}) => Promise<{ object: T; usage?: TokenUsage }>;
+```
 
-### Agent System
+Works with any provider: OpenAI, Anthropic, Google, Mistral, Bedrock, Azure, Ollama, etc.
 
-A composable prompt system that powers conversational AI across platforms:
+### Extraction Pipeline
 
-- **Multi-platform support** — email, chat, SMS, Slack, Discord with platform-aware formatting
-- **Communication intents** — direct (user-facing), mediated (forwarded), observed (CC'd)
-- **Composable modules** — identity, safety, formatting, coverage gaps, COI routing, quotes-vs-policies, conversation memory, and intent-specific instructions
-- **`buildAgentSystemPrompt(ctx)`** — composes all modules into a complete system prompt
-- **Document context builder** — scores and ranks policies/quotes by relevance to a query
-- **Tool definitions** — `tool_use`-compatible schemas for document lookup, COI generation, and coverage comparison
+The `createExtractor` function returns an extraction engine:
 
-### Intent Classification
+1. **Classify** — Determine document type (policy/quote) and line of business
+2. **Plan** — Generate extraction plan using line-specific templates
+3. **Extract** — Dispatch focused extractors in parallel (concurrency-limited, default 2)
+4. **Review** — Check completeness against template requirements (up to 2 review rounds)
+5. **Assemble** — Merge results into final `InsuranceDocument`
 
-Platform-agnostic message classification with `buildClassifyMessagePrompt(platform)` — determines whether an incoming message is insurance-related and suggests an intent.
+```typescript
+const extract = createExtractor({
+  generateText,
+  generateObject,
+  concurrency: 2,           // Parallel extractor limit
+  maxReviewRounds: 2,       // Review loop iterations
+  onTokenUsage: (usage) => {
+    console.log(`${usage.inputTokens} in, ${usage.outputTokens} out`);
+  },
+});
+```
 
-### Insurance Domain Types
+### Document Types
 
-Comprehensive TypeScript type system for the insurance domain:
+Comprehensive TypeScript types for the insurance domain:
 
-- **Document types** — `PolicyDocument`, `QuoteDocument`, and `InsuranceDocument` discriminated union
-- **Platform types** — `Platform`, `CommunicationIntent`, `PlatformConfig`, `AgentContext`
-- **Model types** — `ModelConfig`, `createUniformModelConfig`, `TokenLimits`, `DEFAULT_TOKEN_LIMITS`
+```typescript
+import type {
+  InsuranceDocument,      // PolicyDocument | QuoteDocument
+  PolicyDocument,
+  QuoteDocument,
+  Coverage,
+  Endorsement,
+  Declaration,            // 20+ line types
+  Platform,
+  AgentContext,
+} from "@claritylabs/cl-sdk";
+```
 
 ## API Reference
 
-### Extraction
+### Core Functions
 
 | Function | Description |
 |----------|-------------|
-| `classifyDocumentType(pdf, options)` | Classify document as policy or quote |
-| `extractFromPdf(pdf, options)` | Full policy extraction (passes 1-3) |
-| `extractQuoteFromPdf(pdf, options)` | Full quote extraction (passes 1-2) |
-| `extractSectionsOnly(pdf, metadata, options)` | Retry pass 2 using saved metadata |
-| `applyExtracted(extracted)` | Map extraction JSON to persistence fields |
-| `applyExtractedQuote(extracted)` | Map quote extraction JSON to persistence fields |
+| `createExtractor(config)` | Create extraction engine with callbacks |
+| `extract.extract(pdfBase64, documentId?)` | Run full extraction pipeline |
+| `chunkDocument(text, maxChunkSize?)` | Chunk text for vector storage |
 
-### Options
+### Agent System
 
 ```typescript
-interface ExtractOptions {
-  log?: LogFn;
-  onMetadata?: (raw: string) => Promise<void>;
-  models: ModelConfig;             // required — bring your own models
-  metadataProviderOptions?: ProviderOptions;
-  fallbackProviderOptions?: ProviderOptions;
-  concurrency?: number;            // parallel chunk limit (default: 2)
-  tokenLimits?: TokenLimits;       // override default maxTokens per role
-  onTokenUsage?: (usage: TokenUsage) => void;
-  pdfContentFormat?: "file" | "image";             // default: "file"
-  convertPdfToImages?: ConvertPdfToImagesFn;       // required when pdfContentFormat is "image"
-}
-
-interface ExtractSectionsOptions {
-  log?: LogFn;
-  promptBuilder?: PromptBuilder;
-  models: ModelConfig;             // required — bring your own models
-  fallbackProviderOptions?: ProviderOptions;
-  concurrency?: number;            // parallel chunk limit (default: 2)
-  tokenLimits?: TokenLimits;       // override default maxTokens per role
-  onTokenUsage?: (usage: TokenUsage) => void;
-  pdfContentFormat?: "file" | "image";             // default: "file"
-  convertPdfToImages?: ConvertPdfToImagesFn;       // required when pdfContentFormat is "image"
-}
-
-interface ClassifyOptions {
-  log?: LogFn;
-  models: ModelConfig;             // required — bring your own models
-  tokenLimits?: TokenLimits;       // override default maxTokens per role
-  onTokenUsage?: (usage: TokenUsage) => void;
-  pdfContentFormat?: "file" | "image";             // default: "file"
-  convertPdfToImages?: ConvertPdfToImagesFn;       // required when pdfContentFormat is "image"
-}
-
-// Override default token limits per role (all fields optional)
-interface TokenLimits {
-  classification?: number;  // default: 512
-  metadata?: number;        // default: 16384
-  sections?: number;        // default: 8192
-  sectionsFallback?: number; // default: 16384
-  enrichment?: number;      // default: 4096
-}
-
-interface TokenUsage {
-  inputTokens: number;
-  outputTokens: number;
-}
-
-type ConvertPdfToImagesFn = (
-  pdfBase64: string,
-  startPage: number,
-  endPage: number,
-) => Promise<Array<{ imageBase64: string; mimeType: string }>>;
+import {
+  buildAgentSystemPrompt,
+  buildIdentityPrompt,
+  buildSafetyPrompt,
+  buildCoverageGapPrompt,
+} from "@claritylabs/cl-sdk";
+
+const systemPrompt = buildAgentSystemPrompt({
+  platform: "email",
+  intent: "direct",
+  userName: "John",
+  companyName: "Acme Insurance",
+});
 ```
 
-### PDF Content Formats
-
-| Format | Description | When to use |
-|--------|-------------|-------------|
-| `file` (default) | Send PDF as a native file. Most efficient — no conversion needed. | Most models (Anthropic, Google, OpenAI, Mistral, Bedrock, Azure) |
-| `image` | Convert PDF pages to images via `convertPdfToImages` callback. | Models that don't support native PDF file input |
-
-The SDK defaults to `"file"` — most modern models support native PDF input. If your model doesn't, set `pdfContentFormat: "image"` and provide a `convertPdfToImages` callback. The SDK does not bundle a converter — use whatever library works in your runtime (pdf2pic, mupdf, pdfjs-dist, etc.).
-
-### Rate-Limit Resilience
-
-All model calls automatically retry on rate-limit errors (HTTP 429 or "rate limit" in error message) with exponential backoff — up to 5 retries with delays of 2s, 4s, 8s, 16s, 32s (plus jitter). Non-rate-limit errors are re-thrown immediately.
-
-### Parallel Chunk Extraction
-
-Pass 2 section extraction processes page chunks in parallel with a configurable concurrency limit (default: 2). This balances throughput against rate limits. Sub-chunk retries on truncation are also parallelized.
+### Tool Definitions
 
 ```typescript
-// Track token usage across all passes
-let totalInput = 0, totalOutput = 0;
-
-const { extracted } = await extractFromPdf(pdfBase64, {
-  models: createUniformModelConfig(yourModel),
-  concurrency: 3,
-  onTokenUsage: ({ inputTokens, outputTokens }) => {
-    totalInput += inputTokens;
-    totalOutput += outputTokens;
-  },
-});
-
-console.log(`Total: ${totalInput} input, ${totalOutput} output tokens`);
+import {
+  AGENT_TOOLS,
+  DOCUMENT_LOOKUP_TOOL,
+  COI_GENERATION_TOOL,
+} from "@claritylabs/cl-sdk";
 ```
 
-### Agent
+### PDF Operations
 
-| Function | Description |
-|----------|-------------|
-| `buildAgentSystemPrompt(ctx)` | Full system prompt from `AgentContext` |
-| `buildDocumentContext(docs, query)` | Ranked document context for a query |
-| `buildClassifyMessagePrompt(platform)` | Intent classification prompt |
+```typescript
+import {
+  getAcroFormFields,
+  fillAcroForm,
+  overlayTextOnPdf,
+} from "@claritylabs/cl-sdk";
+```
 
-### PDF Operations
+### Storage
 
-| Function | Description |
-|----------|-------------|
-| `getAcroFormFields(pdfBytes)` | Detect fillable form fields |
-| `fillAcroForm(pdfBytes, mappings)` | Fill and flatten form fields |
-| `overlayTextOnPdf(pdfBytes, overlays)` | Position text on flat PDFs |
+```typescript
+import { SQLiteDocumentStore, SQLiteMemoryStore } from "@claritylabs/cl-sdk/storage/sqlite";
+```
 
 ## Development
 
 ```bash
 npm install
-npm run build      # Build ESM + CJS + types via tsup
+npm run build      # Build ESM + CJS + types
 npm run dev        # Watch mode
-npm run typecheck  # Type check (tsc --noEmit)
+npm run typecheck  # Type check
+npm run test       # Run tests (vitest)
 ```
 
-Pure TypeScript — no framework dependencies. Peer dependencies on `ai` (Vercel AI SDK) and `pdf-lib`. Model-agnostic — bring any provider (`@ai-sdk/anthropic`, `@ai-sdk/openai`, `@ai-sdk/google`, etc.).
+Zero framework dependencies. Peer deps: `pdf-lib`, `zod`. Optional: `better-sqlite3`.
+
+## License
+
+Apache-2.0