@claritylabs/cl-sdk 0.17.0 → 0.18.0

package/README.md

@@ -1,6 +1,6 @@
 # CL-SDK
 
-Open infrastructure for building AI agents that work with insurance. Pure TypeScript, provider-agnostic — bring any LLM, any embedding model, any storage backend.
+Deterministic insurance intelligence primitives for regulated AI agents. Pure TypeScript, provider-agnostic — bring any LLM, any embedding model, any storage backend.
 
 **[Documentation](https://cl-sdk.claritylabs.inc/docs)** | **[npm](https://www.npmjs.com/package/@claritylabs/cl-sdk)** | **[GitHub](https://github.com/claritylabs-inc/cl-sdk)**
 
@@ -12,11 +12,14 @@ npm install @claritylabs/cl-sdk pdf-lib zod
 
 ## What It Does
 
-- **Document Extraction** — Agentic pipeline with 13 focused extractors that turns insurance PDFs into structured data with page-level provenance, quality gates, first-class definitions and covered reasons, and automatic declarations-to-schema promotion (limits, deductibles, locations, broker, loss payees, premium, taxes/fees, summary)
-- **Query Agent** — Citation-backed question answering over stored documents and inbound photos/PDFs/text with sub-question decomposition and grounding verification
-- **Application Processing** — Eight focused agents handle intake — field extraction, auto-fill from prior answers, topic-based question batching, and PDF mapping
-- **Agent System** — Composable prompt modules for building insurance-aware conversational agents across email, chat, SMS, Slack, and Discord
-- **Storage** — DocumentStore and MemoryStore interfaces with SQLite reference implementation
+- **Document Extraction** — Deterministic extraction pipeline with focused model calls that turns insurance PDFs into structured data with page-level provenance, quality gates, first-class definitions and covered reasons, referential coverage resolution, cost-aware formatting, and automatic declarations-to-schema promotion (limits, deductibles, locations, broker, loss payees, premium, taxes/fees, summary)
+- **Source Grounding** — Shared source spans, source chunks, source stores, quoted evidence validation, and deterministic evidence ordering across extraction, query, application, PCE, and case workflows
+- **Query Agent** — Citation-backed question answering over stored documents, source spans, and inbound photos/PDFs/text with sub-question decomposition, bounded retrieval planning, attachment-only reasoning when retrieval is unnecessary, and grounding verification
+- **Application Processing** — Bounded workflows handle intake with deterministic planning — field extraction, prior-answer backfill, context auto-fill, document lookup gating, topic-based question batching, reply parsing, source-backed field provenance, and PDF mapping
+- **Policy Change Endorsements** — PCE intake, evidence collection, missing-info handling, quality gates, execution mode selection, and reviewable submission packets
+- **Case Workflows** — Shared primitives for evidence-backed proposals, missing information, validation issues, stable IDs, and packet artifacts
+- **Agent System** — Composable prompt modules for building insurance-aware agents across email, chat, SMS, Slack, and Discord with human-reviewable behavior
+- **Storage** — DocumentStore, MemoryStore, SourceStore, and ApplicationStore interfaces with reference implementations where appropriate
 
 ## Quick Start
 
@@ -38,9 +41,28 @@ const extractor = createExtractor({
 const result = await extractor.extract(pdfBase64);
 console.log(result.document);     // Typed InsuranceDocument
 console.log(result.chunks);       // DocumentChunk[] for vector storage
+console.log(result.sourceSpans);  // SourceSpan[] when supplied by the host
 console.log(result.reviewReport); // Quality gate results
 ```
 
+## Source Grounding
+
+Source spans are the v1 evidence layer. Build spans from PDF text, OCR, emails, attachments, or structured fields, then pass them into extraction and downstream workflows:
+
+```typescript
+import { buildPageSourceSpans, MemorySourceStore, createExtractor } from "@claritylabs/cl-sdk";
+
+const pageOneText = "..."; // text from your PDF text/OCR pipeline
+const sourceSpans = buildPageSourceSpans([
+  { documentId: "policy-123", sourceKind: "policy_pdf", pageNumber: 1, text: pageOneText },
+]);
+
+const sourceStore = new MemorySourceStore();
+const extractor = createExtractor({ generateText, generateObject, sourceStore });
+
+const result = await extractor.extract(pdfBase64, "policy-123", { sourceSpans });
+```
+
 See the [full documentation](https://cl-sdk.claritylabs.inc/docs) for architecture, provider setup, API reference, and more.
 
 ## Multimodal Querying
@@ -59,6 +81,7 @@ const agent = createQueryAgent({
   generateObject,
   documentStore,
   memoryStore,
+  sourceRetriever,
 });
 
 const result = await agent.query({
@@ -81,16 +104,29 @@ const result = await agent.query({
 });
 ```
 
-The query pipeline first interprets each attachment into structured evidence, then combines that with retrieved chunks, document lookups, and conversation history before answering.
+The query workflow first interprets each attachment into structured evidence, then uses the query classifier to decide whether stored-document retrieval is needed. Simple or attachment-only questions can skip retrieval and reason over the available evidence directly; document-backed questions still retrieve chunks, reason over citations, and run grounding verification. Verification can request targeted retry retrieval for weak sub-answers.
 
 Important: your `generateObject` callback must actually forward multimodal payloads from `providerOptions` to the model request:
 
 - `providerOptions.attachments` for generic image/pdf/text inputs
 - `providerOptions.pdfBase64` for PDF inputs
 - `providerOptions.images` for image inputs
+- `providerOptions.sourceSpans` and `providerOptions.sourceChunks` for source evidence when your host passes them through
 
 If your callback ignores those fields, the model will only see the text prompt.
 
+## Bounded Agentic Workflows
+
+CL-SDK uses deterministic scaffolding with agentic decision points rather than fixed all-tools-all-the-time chains:
+
+- Extraction page mapping and review choose focused follow-up extractors from the live extractor catalog. Definitions and covered reasons can fall back through section extraction when a focused run returns no usable records.
+- Supplementary extraction runs only when page assignments, form inventory, existing extracted text, or review follow-up tasks indicate regulatory, claims, notice, cancellation, or contact facts are likely present.
+- Referential coverage resolution tries cheap local section/form matches first, then uses bounded target-specific actions for declarations, schedules, sections, page-location lookup, or skip.
+- Formatting skips the LLM cleanup pass for plain prose and only formats long or noisy content that looks likely to contain markdown, spacing, list, heading, or table artifacts.
+- Application processing plans optional backfill, context auto-fill, document search, batching, reply parsing, lookup, explanations, and next-batch email generation based on current state.
+
+These gates reduce unnecessary provider calls while preserving reliability for edge cases where additional focused extraction or retrieval is needed.
+
 ## Development
 
 ```bash