@claritylabs/cl-sdk 0.2.0 → 0.5.0

package/README.md

@@ -1,6 +1,8 @@
-[Clarity Labs](https://claritylabs.inc) is an applied AI research lab building the infrastructure for AI to work safely with insurance.
+# CL-SDK
 
-AI agents are already executing complex tasks autonomously across industries, but insurance requires context, safeguards, and systems that don't exist yet. CL-0 SDK fills that gap: a shared intelligence layer that any product or agent can import to understand, reason about, and act on insurance.
+[Clarity Labs](https://claritylabs.inc) allows insurers to understand their clients as well as they know themselves. A better understanding of clients means insurers can automate servicing to reduce costs and identify coverage gaps to cross-sell products.
+
+CL-SDK is the open infrastructure layer that makes this possible — a pure TypeScript library for extracting, reasoning about, and acting on insurance documents. Provider-agnostic by design: bring any LLM, any embedding model, any storage backend.
 
 ## Installation
 
@@ -10,276 +12,568 @@ npm install @claritylabs/cl-sdk
 
 ### Peer Dependencies
 
-CL-0 SDK requires the [Vercel AI SDK](https://sdk.vercel.ai) and pdf-lib:
+```bash
+npm install pdf-lib zod
+```
 
+Optional (for SQLite storage):
 ```bash
-npm install ai pdf-lib
+npm install better-sqlite3
 ```
 
-Then install a provider package for your model of choice:
+## Quick Start
+
+### Document Extraction
 
-```bash
-# Anthropic
-npm install @ai-sdk/anthropic
+CL-SDK extracts structured data from insurance PDFs using a multi-agent pipeline. You provide two callback functions — `generateText` and `generateObject` — and the SDK handles the rest:
 
-# OpenAI
-npm install @ai-sdk/openai
+```typescript
+import { createExtractor } from "@claritylabs/cl-sdk";
 
-# Google
-npm install @ai-sdk/google
+const extractor = createExtractor({
+  generateText: async ({ prompt, system, maxTokens }) => {
+    // Wrap your preferred LLM provider
+    const result = await yourProvider.generate({ prompt, system, maxTokens });
+    return { text: result.text, usage: result.usage };
+  },
+  generateObject: async ({ prompt, system, schema, maxTokens }) => {
+    // schema is a Zod schema — use it for structured output
+    const result = await yourProvider.generateStructured({ prompt, system, schema, maxTokens });
+    return { object: result.object, usage: result.usage };
+  },
+});
+
+const pdfBase64 = "..."; // base64-encoded insurance PDF
+const result = await extractor.extract(pdfBase64);
+console.log(result.document); // Typed InsuranceDocument (policy or quote)
+console.log(result.chunks);   // DocumentChunk[] ready for vector storage
 ```
 
-## Quick Start
+### With PDF-to-Image Conversion
+
+For providers that don't support native PDF input (e.g., OpenAI):
+
+```typescript
+const extractor = createExtractor({
+  generateText: /* ... */,
+  generateObject: /* ... */,
+  convertPdfToImages: async (pdfBase64, startPage, endPage) => {
+    // Convert PDF pages to images using your preferred library
+    return [{ imageBase64: "...", mimeType: "image/png" }]; // one per page
+  },
+});
+```
 
-### Uniform Model (same model for all passes)
+## Architecture
+
+### Provider-Agnostic Callbacks
+
+CL-SDK has **zero framework dependencies**. All LLM interaction happens through two callback types:
 
 ```typescript
-import { createAnthropic } from "@ai-sdk/anthropic";
-import { extractFromPdf, createUniformModelConfig, applyExtracted } from "@claritylabs/cl-sdk";
+type GenerateText = (params: {
+  prompt: string;
+  system?: string;
+  maxTokens: number;
+  providerOptions?: Record<string, unknown>;
+}) => Promise<{ text: string; usage?: { inputTokens: number; outputTokens: number } }>;
+
+type GenerateObject<T> = (params: {
+  prompt: string;
+  system?: string;
+  schema: ZodSchema<T>;
+  maxTokens: number;
+  providerOptions?: Record<string, unknown>;
+}) => Promise<{ object: T; usage?: { inputTokens: number; outputTokens: number } }>;
+```
+
+Works with any provider: Anthropic, OpenAI, Google, Mistral, Bedrock, Azure, Ollama, etc. You write the adapter once; the SDK calls it throughout the pipeline.
+
+### Extraction Pipeline
+
+The extraction system uses a **coordinator/worker pattern** — a coordinator agent plans the work, specialized extractor agents execute in parallel, and a review loop ensures completeness.
+
+```
+┌─────────────┐     ┌─────────────┐     ┌──────────────────────┐
+│  1. CLASSIFY │────▶│  2. PLAN    │────▶│  3. EXTRACT (parallel)│
+│              │     │             │     │                      │
+│  Document    │     │  Select     │     │  Run focused         │
+│  type, line  │     │  template,  │     │  extractors against  │
+│  of business │     │  assign     │     │  assigned page       │
+│              │     │  extractors │     │  ranges              │
+│              │     │  to pages   │     │                      │
+└─────────────┘     └─────────────┘     └──────────┬───────────┘
+                                                   │
+                    ┌─────────────┐     ┌──────────▼───────────┐
+                    │ 5. ASSEMBLE │◀────│  4. REVIEW           │
+                    │             │     │                      │
+                    │  Merge all  │     │  Check completeness  │
+                    │  results,   │     │  against template,   │
+                    │  validate,  │     │  dispatch follow-up  │
+                    │  chunk      │     │  extractors for gaps │
+                    └─────────────┘     └──────────────────────┘
+```
+
+#### Phase 1: Classify
+
+The coordinator sends the document to `generateObject` with the `ClassifyResultSchema`. It determines:
+- **Document type** — policy or quote
+- **Policy types** — one or more lines of business (e.g., `general_liability`, `workers_comp`)
+- **Confidence score**
 
-const anthropic = createAnthropic();
-const pdfBase64 = "..."; // base64-encoded PDF
+#### Phase 2: Plan
 
-const { extracted } = await extractFromPdf(pdfBase64, {
-  models: createUniformModelConfig(anthropic("claude-sonnet-4-6")),
+Based on the classification, the coordinator selects a **line-of-business template** (e.g., `workers_comp`, `cyber`, `homeowners_ho3`) that defines expected sections and page hints. It then generates an **extraction plan** — a list of tasks that map specific extractors to page ranges within the PDF.
+
+#### Phase 3: Extract
+
+Focused extractor agents are dispatched **in parallel** (concurrency-limited, default 2). Each extractor targets a specific data domain against its assigned page range. The 11 extractor types are:
+
+| Extractor | What It Extracts |
+|-----------|-----------------|
+| `carrier_info` | Carrier name, NAIC, AM Best rating, MGA, underwriter, broker |
+| `named_insured` | Insured name, DBA, address, entity type, FEIN, SIC/NAICS |
+| `declarations` | Line-specific structured declarations (varies by policy type) |
+| `coverage_limits` | Coverage names, limits, deductibles, forms, triggers |
+| `endorsements` | Form numbers, titles, types, content, affected parties |
+| `exclusions` | Exclusion titles, content, applicability |
+| `conditions` | Duties after loss, cancellation, other insurance, etc. |
+| `premium_breakdown` | Premium amounts, taxes, fees, payment plans, rating basis |
+| `loss_history` | Loss runs, claim records, experience modification |
+| `supplementary` | Regulatory context, contacts, TPA, claims contacts |
+| `sections` | Raw section content (fallback for unmatched sections) |
+
+Each extractor writes its results to an in-memory `Map`. Results accumulate across all extractors.
+
+#### Phase 4: Review
+
+After initial extraction, a review loop (up to `maxReviewRounds`, default 2) checks completeness against the template's expected sections. If gaps are found, additional extractor tasks are dispatched to fill missing data. This iterative refinement ensures comprehensive extraction.
+
+#### Phase 5: Assemble
+
+All extractor results are merged into a final validated `InsuranceDocument`, then chunked into `DocumentChunk[]` for vector storage. Chunks are deterministically IDed as `${documentId}:${type}:${index}`.
+
+### Configuration
+
+```typescript
+const extractor = createExtractor({
+  // Required: LLM callbacks
+  generateText,
+  generateObject,
+
+  // Optional: PDF vision mode
+  convertPdfToImages: async (pdfBase64, startPage, endPage) => [...],
+
+  // Optional: storage backends
+  documentStore,  // Persist extracted documents
+  memoryStore,    // Vector search over chunks + conversation history
+
+  // Optional: tuning
+  concurrency: 2,        // Max parallel extractors (default: 2)
+  maxReviewRounds: 2,    // Review loop iterations (default: 2)
+
+  // Optional: observability
+  onTokenUsage: (usage) => console.log(`${usage.inputTokens} in, ${usage.outputTokens} out`),
+  onProgress: (message) => console.log(message),
+  log: async (message) => logger.info(message),
+  providerOptions: {},   // Passed through to every LLM call
 });
-const fields = applyExtracted(extracted);
 ```
 
-### Any Provider
+### Line-of-Business Templates
+
+Templates define what the extraction pipeline expects for each policy type. Each template specifies expected sections, page hints, and required vs. optional fields.
+
+**Personal lines:** homeowners (HO-3, HO-5), renters (HO-4), condo (HO-6), dwelling fire, personal auto, personal umbrella, personal inland marine, flood (NFIP + private), earthquake, watercraft, recreational vehicle, farm/ranch, mobile home
+
+**Commercial lines:** general liability, commercial property, commercial auto, workers' comp, umbrella/excess, professional liability, cyber, directors & officers, crime/fidelity
+
+## Storage
+
+CL-SDK defines two storage interfaces (`DocumentStore` and `MemoryStore`) and ships a reference SQLite implementation. You can implement these interfaces with any backend.
+
+### DocumentStore
+
+CRUD for extracted `InsuranceDocument` objects:
+
+```typescript
+interface DocumentStore {
+  save(doc: InsuranceDocument): Promise<void>;
+  get(id: string): Promise<InsuranceDocument | null>;
+  query(filters: DocumentFilters): Promise<InsuranceDocument[]>;
+  delete(id: string): Promise<void>;
+}
+```
+
+Filters support: `type` (policy/quote), `carrier` (fuzzy), `insuredName` (fuzzy), `policyNumber` (exact), `quoteNumber` (exact).
 
-CL-0 SDK is provider-agnostic — PDFs are sent as native files by default, which most providers support (Anthropic, Google, OpenAI, Mistral, Bedrock, Azure):
+### MemoryStore
+
+Vector-searchable storage for document chunks and conversation history. Requires an `EmbedText` callback for generating embeddings:
 
 ```typescript
-import { createOpenAI } from "@ai-sdk/openai";
-import { extractFromPdf, createUniformModelConfig } from "@claritylabs/cl-sdk";
+type EmbedText = (text: string) => Promise<number[]>;
 
-const openai = createOpenAI();
-const { extracted } = await extractFromPdf(pdfBase64, {
-  models: createUniformModelConfig(openai("gpt-4o")),
-});
+interface MemoryStore {
+  // Document chunks with embeddings
+  addChunks(chunks: DocumentChunk[]): Promise<void>;
+  search(query: string, options?: { limit?: number; filter?: ChunkFilter }): Promise<DocumentChunk[]>;
+
+  // Conversation turns with embeddings
+  addTurn(turn: ConversationTurn): Promise<void>;
+  getHistory(conversationId: string, options?: { limit?: number }): Promise<ConversationTurn[]>;
+  searchHistory(query: string, conversationId?: string): Promise<ConversationTurn[]>;
+}
 ```
 
-If your model doesn't support native PDF input, set `pdfContentFormat: "image"` and provide a `convertPdfToImages` callback. The SDK does not bundle a converter — use whatever library works in your runtime (pdf2pic, mupdf, pdfjs-dist, etc.):
+Search uses **cosine similarity** over embeddings to find semantically relevant chunks or conversation turns. Embedding failures are non-fatal — chunks are still stored, just not searchable by vector.
+
+### SQLite Reference Implementation
 
 ```typescript
-const { extracted } = await extractFromPdf(pdfBase64, {
-  models: createUniformModelConfig(yourModel),
-  pdfContentFormat: "image",
-  convertPdfToImages: async (pdfBase64, startPage, endPage) => {
-    // Return one { imageBase64, mimeType } per page
-    return pages;
+import { createSqliteStore } from "@claritylabs/cl-sdk/storage/sqlite";
+
+const store = createSqliteStore({
+  path: "./cl-sdk.db",
+  embed: async (text) => {
+    // Your embedding function (OpenAI, Cohere, local model, etc.)
+    return await yourEmbeddingProvider.embed(text);
   },
 });
-```
 
-### Fine-Grained Model Config
+// Use with extractor
+const extractor = createExtractor({
+  generateText,
+  generateObject,
+  documentStore: store.documents,
+  memoryStore: store.memory,
+});
 
-Assign different models per pipeline role — use a fast model for classification/sections and a capable model for metadata/fallback:
+// Or use standalone
+await store.documents.save(document);
+const results = await store.memory.search("what is the deductible?", { limit: 5 });
 
-```typescript
-import { createAnthropic } from "@ai-sdk/anthropic";
-import { extractFromPdf, type ModelConfig } from "@claritylabs/cl-sdk";
-
-const anthropic = createAnthropic();
-const models: ModelConfig = {
-  classification: anthropic("claude-haiku-4-5-20251001"),  // fast, cheap
-  metadata: anthropic("claude-sonnet-4-6"),                // capable
-  sections: anthropic("claude-haiku-4-5-20251001"),        // fast, cheap
-  sectionsFallback: anthropic("claude-sonnet-4-6"),        // capable (fallback)
-  enrichment: anthropic("claude-haiku-4-5-20251001"),      // fast, cheap
-};
-
-const { extracted } = await extractFromPdf(pdfBase64, { models });
+// Clean up
+store.close();
 ```
 
-### Mixed Providers
+## Agent System
+
+CL-SDK includes a composable prompt system for building insurance-aware AI agents. The `buildAgentSystemPrompt` function assembles modular prompt segments based on the agent's context:
 
 ```typescript
-import { createAnthropic } from "@ai-sdk/anthropic";
-import { createOpenAI } from "@ai-sdk/openai";
-import { extractFromPdf, type ModelConfig } from "@claritylabs/cl-sdk";
-
-const anthropic = createAnthropic();
-const openai = createOpenAI();
-
-const models: ModelConfig = {
-  classification: openai("gpt-4o-mini"),
-  metadata: anthropic("claude-sonnet-4-6"),
-  sections: openai("gpt-4o-mini"),
-  sectionsFallback: anthropic("claude-sonnet-4-6"),
-  enrichment: openai("gpt-4o-mini"),
-};
-
-const { extracted } = await extractFromPdf(pdfBase64, { models });
+import { buildAgentSystemPrompt } from "@claritylabs/cl-sdk";
+
+const systemPrompt = buildAgentSystemPrompt({
+  platform: "email",       // email | chat | sms | slack | discord
+  intent: "direct",        // direct | mediated | observed
+  userName: "John",
+  companyName: "Acme Insurance",
+});
 ```
 
-## What's Inside
+### Prompt Modules
 
-### Document Extraction Pipeline
+The system prompt is composed from these modules:
 
-A multi-pass system that turns insurance PDFs into structured, queryable data:
+| Module | Purpose |
+|--------|---------|
+| **identity** | Agent role, company context, professional persona |
+| **intent** | Behavioral rules based on platform and interaction mode |
+| **formatting** | Output formatting rules (markdown for chat, plaintext for email/SMS) |
+| **safety** | Security guardrails, prompt injection resistance, data handling |
+| **coverage-gaps** | Coverage gap disclosure rules (only in mediated/observed mode) |
+| **coi-routing** | Certificate of Insurance request handling |
+| **quotes-policies** | Guidance for distinguishing quotes vs. active policies |
+| **conversation-memory** | Context about conversation history and document retrieval |
 
-- **Pass 0 — Classification**: Determines whether a document is a policy or a quote. Returns document type, confidence score, and supporting signals.
-- **Pass 1 — Metadata Extraction**: Extracts high-level metadata — carrier, policy/quote number, dates, premium, insured name, coverage table with limits and deductibles. Includes an early persistence callback (`onMetadata`) so metadata is saved immediately, surviving downstream failures.
-- **Pass 2 — Section Extraction**: Splits the document into page chunks (starting at 15 pages) and extracts structured sections in parallel (concurrency-limited, default 2). All model calls automatically retry on rate-limit errors with exponential backoff. Adaptive fallback: if a chunk's output is truncated (JSON parse failure), it re-splits into smaller chunks (10, then 5 pages), and escalates to the fallback model. Results are merged across chunks.
-- **Pass 3 — Enrichment**: A non-fatal pass that parses raw text into structured supplementary fields — regulatory context, complaint contacts, costs and fees, claims contacts.
+### Message Intent Classification
 
-For quotes specifically, the pipeline also extracts premium breakdowns, subjectivities (conditions that must be met before binding), and underwriting conditions.
+Classify incoming messages to route them appropriately:
 
-### Application Processing
+```typescript
+import { buildClassifyMessagePrompt } from "@claritylabs/cl-sdk";
 
-End-to-end insurance application handling — from blank PDF to filled form:
+const prompt = buildClassifyMessagePrompt("email");
+// Returns classification prompt for intents:
+// policy_question, coi_request, renewal_inquiry, claim_report,
+// coverage_shopping, general, unrelated
+```
 
-- **Application detection** — classifies whether a PDF is an insurance application form
-- **Field extraction** — reads every fillable field as structured data (text, numeric, currency, date, yes/no, table, and declaration fields)
-- **Auto-fill** — matches extracted fields against known business context to pre-populate answers
-- **Question batching** — organizes unfilled fields into topic-based batches for emailing the insured
-- **Answer parsing** — parses free-text replies back into structured field values
-- **PDF filling** — maps answers back onto the original PDF, supporting both AcroForm and flat PDF overlay
+## Application Processing Pipeline
 
-### Agent System
+The application pipeline processes insurance applications through an agentic coordinator/worker system — small focused agents handle classification, field extraction, auto-fill, question batching, reply routing, and PDF mapping. Supports persistent state and vector-based answer backfill from prior applications.
 
-A composable prompt system that powers conversational AI across platforms:
+### Quick Start
 
-- **Multi-platform support** — email, chat, SMS, Slack, Discord with platform-aware formatting
-- **Communication intents** — direct (user-facing), mediated (forwarded), observed (CC'd)
-- **Composable modules** — identity, safety, formatting, coverage gaps, COI routing, quotes-vs-policies, conversation memory, and intent-specific instructions
-- **`buildAgentSystemPrompt(ctx)`** — composes all modules into a complete system prompt
-- **Document context builder** — scores and ranks policies/quotes by relevance to a query
-- **Tool definitions** — `tool_use`-compatible schemas for document lookup, COI generation, and coverage comparison
+```typescript
+import { createApplicationPipeline } from "@claritylabs/cl-sdk";
+
+const pipeline = createApplicationPipeline({
+  generateText,
+  generateObject,
+  applicationStore,      // persistent state storage
+  documentStore,         // for policy/quote lookups during auto-fill
+  memoryStore,           // for vector-based answer backfill
+  orgContext: [           // business context for auto-fill
+    { key: "company_name", value: "Acme Corp", category: "company_info" },
+    { key: "company_address", value: "123 Main St", category: "company_info" },
+  ],
+});
 
-### Intent Classification
+// Process a new application PDF
+const { state } = await pipeline.processApplication({
+  pdfBase64: "...",
+  applicationId: "app-123",
+});
+// state.fields → extracted fields, some already auto-filled
+// state.batches → question batches ready for user collection
 
-Platform-agnostic message classification with `buildClassifyMessagePrompt(platform)` — determines whether an incoming message is insurance-related and suggests an intent.
+// Generate email for current batch
+const { text: emailBody } = await pipeline.generateCurrentBatchEmail("app-123", {
+  companyName: "Acme Corp",
+});
 
-### Insurance Domain Types
+// Process user's reply
+const { state: updated, fieldsFilled, responseText } = await pipeline.processReply({
+  applicationId: "app-123",
+  replyText: "1. Yes\n2. $1,000,000\n3. Check our website for revenue",
+});
+```
 
-Comprehensive TypeScript type system for the insurance domain:
+### Pipeline Phases
 
-- **Document types** — `PolicyDocument`, `QuoteDocument`, and `InsuranceDocument` discriminated union
-- **Platform types** — `Platform`, `CommunicationIntent`, `PlatformConfig`, `AgentContext`
-- **Model types** — `ModelConfig`, `createUniformModelConfig`, `TokenLimits`, `DEFAULT_TOKEN_LIMITS`
+```
+┌─────────────┐     ┌──────────────┐     ┌─────────────────────┐
+│ 1. CLASSIFY  │────>│ 2. EXTRACT   │────>│ 3. BACKFILL +       │
+│              │     │    FIELDS    │     │    AUTO-FILL         │
+│ Is this an   │     │              │     │    (parallel)        │
+│ application? │     │ All fillable │     │                     │
+│              │     │ fields as    │     │ • vector backfill   │
+│              │     │ structured   │     │ • context auto-fill │
+│              │     │ data         │     │ • document search   │
+└──────────────┘     └──────────────┘     └──────────┬──────────┘
+                                                     │
+                     ┌──────────────┐     ┌──────────v──────────┐
+                     │ REPLY LOOP   │<────│ 4. BATCH QUESTIONS  │
+                     │              │     │                     │
+                     │ Route intent │     │ Group unfilled      │
+                     │ Parse answers│     │ fields by topic     │
+                     │ Handle lookup│     │ Generate emails     │
+                     │ Explain field│     │                     │
+                     └──────┬───────┘     └─────────────────────┘
+                            │
+                     ┌──────v───────┐
+                     │ 5. CONFIRM + │
+                     │    MAP PDF   │
+                     └──────────────┘
+```
 
-## API Reference
+### Focused Agents (8 types)
 
-### Extraction
+| Agent | Task | Model Size |
+|-------|------|-----------|
+| `classifier` | Detect if PDF is an application | Tiny |
+| `field-extractor` | Extract all form fields | Medium |
+| `auto-filler` | Match fields to business context | Small |
+| `batcher` | Group fields into topic batches | Small |
+| `reply-router` | Classify reply intent | Tiny |
+| `answer-parser` | Extract answers from replies | Small |
+| `lookup-filler` | Fill from policy/record lookups | Small |
+| `email-generator` | Generate professional batch emails | Small |
 
-| Function | Description |
-|----------|-------------|
-| `classifyDocumentType(pdf, options)` | Classify document as policy or quote |
-| `extractFromPdf(pdf, options)` | Full policy extraction (passes 1-3) |
-| `extractQuoteFromPdf(pdf, options)` | Full quote extraction (passes 1-2) |
-| `extractSectionsOnly(pdf, metadata, options)` | Retry pass 2 using saved metadata |
-| `applyExtracted(extracted)` | Map extraction JSON to persistence fields |
-| `applyExtractedQuote(extracted)` | Map quote extraction JSON to persistence fields |
+### Vector-Based Answer Backfill
 
-### Options
+The `BackfillProvider` interface enables searching prior application answers and extracted document data to pre-fill new applications:
 
 ```typescript
-interface ExtractOptions {
-  log?: LogFn;
-  onMetadata?: (raw: string) => Promise<void>;
-  models: ModelConfig;             // required — bring your own models
-  metadataProviderOptions?: ProviderOptions;
-  fallbackProviderOptions?: ProviderOptions;
-  concurrency?: number;            // parallel chunk limit (default: 2)
-  tokenLimits?: TokenLimits;       // override default maxTokens per role
-  onTokenUsage?: (usage: TokenUsage) => void;
-  pdfContentFormat?: "file" | "image";             // default: "file"
-  convertPdfToImages?: ConvertPdfToImagesFn;       // required when pdfContentFormat is "image"
+interface BackfillProvider {
+  searchPriorAnswers(
+    fields: { id: string; label: string; section: string; fieldType: string }[],
+    options?: { limit?: number },
+  ): Promise<PriorAnswer[]>;
 }
+```
 
-interface ExtractSectionsOptions {
-  log?: LogFn;
-  promptBuilder?: PromptBuilder;
-  models: ModelConfig;             // required — bring your own models
-  fallbackProviderOptions?: ProviderOptions;
-  concurrency?: number;            // parallel chunk limit (default: 2)
-  tokenLimits?: TokenLimits;       // override default maxTokens per role
-  onTokenUsage?: (usage: TokenUsage) => void;
-  pdfContentFormat?: "file" | "image";             // default: "file"
-  convertPdfToImages?: ConvertPdfToImagesFn;       // required when pdfContentFormat is "image"
-}
+This runs in parallel with context-based auto-fill, so the pipeline fills as many fields as possible before asking the user anything.
 
-interface ClassifyOptions {
-  log?: LogFn;
-  models: ModelConfig;             // required — bring your own models
-  tokenLimits?: TokenLimits;       // override default maxTokens per role
-  onTokenUsage?: (usage: TokenUsage) => void;
-  pdfContentFormat?: "file" | "image";             // default: "file"
-  convertPdfToImages?: ConvertPdfToImagesFn;       // required when pdfContentFormat is "image"
-}
+### Application Prompts (for advanced use)
 
-// Override default token limits per role (all fields optional)
-interface TokenLimits {
-  classification?: number;  // default: 512
-  metadata?: number;        // default: 16384
-  sections?: number;        // default: 8192
-  sectionsFallback?: number; // default: 16384
-  enrichment?: number;      // default: 4096
-}
+The individual prompt functions are still exported for custom pipelines:
 
-interface TokenUsage {
-  inputTokens: number;
-  outputTokens: number;
-}
+```typescript
+import {
+  buildFieldExtractionPrompt,
+  buildAutoFillPrompt,
+  buildQuestionBatchPrompt,
+  buildAnswerParsingPrompt,
+  buildConfirmationSummaryPrompt,
+  buildBatchEmailGenerationPrompt,
+  buildReplyIntentClassificationPrompt,
+  buildFieldExplanationPrompt,
+  buildFlatPdfMappingPrompt,
+  buildAcroFormMappingPrompt,
+  buildLookupFillPrompt,
+} from "@claritylabs/cl-sdk";
+```
+
+## Query Agent Pipeline
+
+The query agent answers user questions against stored documents with citation-backed provenance. It mirrors the extraction pipeline's coordinator/worker pattern: a classifier decomposes questions, retrievers pull evidence in parallel, reasoners answer from evidence only, and a verifier checks grounding.
+
+### Quick Start
+
+```typescript
+import { createQueryAgent } from "@claritylabs/cl-sdk";
+
+const agent = createQueryAgent({
+  generateText,
+  generateObject,
+  documentStore,    // where extracted documents are stored
+  memoryStore,      // where document chunks + conversation history live
+});
 
-type ConvertPdfToImagesFn = (
-  pdfBase64: string,
-  startPage: number,
-  endPage: number,
-) => Promise<Array<{ imageBase64: string; mimeType: string }>>;
+const result = await agent.query({
+  question: "What is the deductible on our GL policy?",
+  conversationId: "conv-123",
+});
+
+console.log(result.answer);     // Natural language answer
+console.log(result.citations);  // Source references with exact quotes
+console.log(result.confidence); // 0-1 confidence score
 ```
 
-### PDF Content Formats
+### Pipeline Phases
+
+```
+┌─────────────┐     ┌──────────────┐     ┌────────────────────┐
+│ 1. CLASSIFY  │────>│ 2. RETRIEVE  │────>│ 3. REASON          │
+│              │     │  (parallel)  │     │    (parallel)       │
+│ Intent +     │     │              │     │                    │
+│ sub-question │     │ chunk search │     │ Answer each sub-Q  │
+│ decomposition│     │ doc lookup   │     │ from evidence only │
+│              │     │ conv history │     │                    │
+└──────────────┘     └──────────────┘     └─────────┬──────────┘
+                                                    │
+                     ┌──────────────┐     ┌─────────v──────────┐
+                     │ 5. RESPOND   │<────│ 4. VERIFY          │
+                     │              │     │                    │
+                     │ Format with  │     │ Grounding check    │
+                     │ citations,   │     │ Consistency check  │
+                     │ store turn   │     │ Completeness check │
+                     └──────────────┘     └────────────────────┘
+```
 
-| Format | Description | When to use |
-|--------|-------------|-------------|
-| `file` (default) | Send PDF as a native file. Most efficient — no conversion needed. | Most models (Anthropic, Google, OpenAI, Mistral, Bedrock, Azure) |
-| `image` | Convert PDF pages to images via `convertPdfToImages` callback. | Models that don't support native PDF file input |
+**Phase 1 — Classify:** Determines intent (`policy_question`, `coverage_comparison`, `document_search`, `claims_inquiry`, `general_knowledge`) and decomposes complex questions into atomic sub-questions. Each sub-question specifies which chunk types and document filters to use for retrieval.
 
-The SDK defaults to `"file"` — most modern models support native PDF input. If your model doesn't, set `pdfContentFormat: "image"` and provide a `convertPdfToImages` callback. The SDK does not bundle a converter — use whatever library works in your runtime (pdf2pic, mupdf, pdfjs-dist, etc.).
+**Phase 2 — Retrieve (parallel):** For each sub-question, a retriever searches chunk embeddings, does structured document lookups, and pulls conversation history — all in parallel. Returns ranked evidence items.
 
-### Rate-Limit Resilience
+**Phase 3 — Reason (parallel):** For each sub-question, a reasoner receives only the retrieved evidence (never the full document) and produces a sub-answer with citations. Intent-specific prompts guide reasoning (e.g., coverage questions get prompts tuned for interpreting limits and endorsements).
 
-All model calls automatically retry on rate-limit errors (HTTP 429 or "rate limit" in error message) with exponential backoff — up to 5 retries with delays of 2s, 4s, 8s, 16s, 32s (plus jitter). Non-rate-limit errors are re-thrown immediately.
+**Phase 4 — Verify:** The verifier checks that every claim is grounded in a citation, sub-answers don't contradict each other, and no evidence was overlooked. If issues are found, it can trigger re-retrieval with broader context.
 
-### Parallel Chunk Extraction
+**Phase 5 — Respond:** Merges verified sub-answers into a single natural-language response with inline citations (`[1]`, `[2]`), deduplicates references, and stores the exchange as conversation turns.
 
-Pass 2 section extraction processes page chunks in parallel with a configurable concurrency limit (default: 2). This balances throughput against rate limits. Sub-chunk retries on truncation are also parallelized.
+### Configuration
 
 ```typescript
-// Track token usage across all passes
-let totalInput = 0, totalOutput = 0;
-
-const { extracted } = await extractFromPdf(pdfBase64, {
-  models: createUniformModelConfig(yourModel),
-  concurrency: 3,
-  onTokenUsage: ({ inputTokens, outputTokens }) => {
-    totalInput += inputTokens;
-    totalOutput += outputTokens;
-  },
+const agent = createQueryAgent({
+  // Required
+  generateText,
+  generateObject,
+  documentStore,
+  memoryStore,
+
+  // Optional: tuning
+  concurrency: 3,        // max parallel retrievers/reasoners (default: 3)
+  maxVerifyRounds: 1,    // verification loop iterations (default: 1)
+  retrievalLimit: 10,    // max evidence items per sub-question (default: 10)
+
+  // Optional: observability
+  onTokenUsage: (usage) => console.log(`${usage.inputTokens} in, ${usage.outputTokens} out`),
+  onProgress: (message) => console.log(message),
+  log: async (message) => logger.info(message),
+  providerOptions: {},
 });
+```
+
+### Citations
+
+Every factual claim in the answer references its source:
+
+```typescript
+interface Citation {
+  index: number;         // [1], [2], etc.
+  chunkId: string;       // e.g. "doc-123:coverage:2"
+  documentId: string;
+  documentType?: "policy" | "quote";
+  field?: string;        // e.g. "coverages[0].deductible"
+  quote: string;         // exact text from source
+  relevance: number;     // 0-1 similarity score
+}
+```
+
+## PDF Operations
+
+```typescript
+import {
+  extractPageRange,    // Extract specific pages from a PDF
+  getPdfPageCount,     // Get total page count
+  getAcroFormFields,   // Enumerate form fields (text, checkbox, dropdown, radio)
+  fillAcroForm,        // Fill and flatten AcroForm fields
+  overlayTextOnPdf,    // Overlay text at coordinates on flat PDFs
+} from "@claritylabs/cl-sdk";
+```
+
+## Tool Definitions
+
+Claude `tool_use`-compatible schemas for agent integrations:
 
-console.log(`Total: ${totalInput} input, ${totalOutput} output tokens`);
+```typescript
+import {
+  AGENT_TOOLS,              // All tools as an array
+  DOCUMENT_LOOKUP_TOOL,     // Search/retrieve policies and quotes
+  COI_GENERATION_TOOL,      // Generate Certificates of Insurance
+  COVERAGE_COMPARISON_TOOL, // Compare coverages across documents
+} from "@claritylabs/cl-sdk";
 ```
 
-### Agent
+These are schema-only definitions (input schemas + descriptions). You provide the implementations that call your storage and PDF layers.
+
+## Document Types
 
-| Function | Description |
-|----------|-------------|
-| `buildAgentSystemPrompt(ctx)` | Full system prompt from `AgentContext` |
-| `buildDocumentContext(docs, query)` | Ranked document context for a query |
-| `buildClassifyMessagePrompt(platform)` | Intent classification prompt |
+All types are derived from Zod schemas, providing both runtime validation and TypeScript types:
 
-### PDF Operations
+```typescript
+import type {
+  InsuranceDocument,   // PolicyDocument | QuoteDocument (discriminated union)
+  PolicyDocument,      // Extracted policy with all enrichments
+  QuoteDocument,       // Extracted quote with subjectivities, premium breakdown
+  Coverage,            // Coverage name, limits, deductibles, form
+  EnrichedCoverage,    // Coverage + additional metadata
+  Endorsement,         // Form number, title, type, content
+  Exclusion,           // Title, content, applicability
+  Condition,           // Type, title, content
+  Declaration,         // Line-specific declarations (19 types)
+  Platform,            // email | chat | sms | slack | discord
+  AgentContext,        // Platform + intent + user/company context
+} from "@claritylabs/cl-sdk";
+```
 
-| Function | Description |
-|----------|-------------|
-| `getAcroFormFields(pdfBytes)` | Detect fillable form fields |
-| `fillAcroForm(pdfBytes, mappings)` | Fill and flatten form fields |
-| `overlayTextOnPdf(pdfBytes, overlays)` | Position text on flat PDFs |
+### Supported Policy Types
+
+42 policy types across personal and commercial lines — including general liability, commercial property, workers' comp, cyber, D&O, homeowners (HO-3/HO-5/HO-4/HO-6), personal auto, flood (NFIP + private), earthquake, and more.
+
+## Core Utilities
+
+```typescript
+import {
+  withRetry,       // Exponential backoff with jitter (5 retries, 2–32s) for rate limits
+  pLimit,          // Concurrency limiter for parallel async tasks
+  sanitizeNulls,   // Recursively convert null → undefined (for database compatibility)
+  stripFences,     // Remove markdown code fences from LLM JSON responses
+} from "@claritylabs/cl-sdk";
+```
 
 ## Development
 
@@ -290,4 +584,8 @@ npm run dev        # Watch mode
 npm run typecheck  # Type check (tsc --noEmit)
 ```
 
-Pure TypeScript — no framework dependencies. Peer dependencies on `ai` (Vercel AI SDK) and `pdf-lib`. Model-agnostic — bring any provider (`@ai-sdk/anthropic`, `@ai-sdk/openai`, `@ai-sdk/google`, etc.).
+Zero framework dependencies. Peer deps: `pdf-lib`, `zod`. Optional: `better-sqlite3`.
+
+## License
+
+Apache-2.0