ai-database 2.1.3 → 2.3.0

package/README.md

@@ -1,219 +1,389 @@
 # ai-database
 
-**AI-generated data shouldn't be disconnected from your schema.**
+![Stability: Stable](https://img.shields.io/badge/stability-stable-green)
 
-You write one line of AI code. It generates a user. Where does it go? Does it have a company? Does it match your existing customers? Traditional databases don't know. They can't reason about relationships. They can't cascade.
+**AI hallucinates. Your database shouldn't.**
 
-**ai-database can.**
+When AI generates a "Software Developer" for your customer profile, does it match your existing O\*NET occupation data? Does "Enterprise SaaS" connect to your NAICS industry codes? Traditional approaches fragment context—AI juggles content creation and referential integrity simultaneously, producing plausible-sounding but disconnected data.
+
+**ai-database grounds AI generation against your domain.**
 
 ```typescript
 import { DB } from 'ai-database'
 
 const { db } = DB({
-  Lead: { name: 'string', company: 'Company.leads', score: 'number' },
-  Company: { name: 'string', industry: 'string' }
+  IdealCustomerProfile: {
+    as: 'Who are they? <~Occupation',      // Ground against O*NET occupations
+    at: 'Where do they work? <~Industry',  // Ground against NAICS industries
+    are: 'What are they doing? <~Task',    // Ground against O*NET tasks
+  },
+  Occupation: { title: 'string', description: 'string' },
+  Industry: { name: 'string', naicsCode: 'string' },
+  Task: { name: 'string' },
+})
+
+// Seed reference data from O*NET, NAICS, etc.
+await db.Occupation.create({ title: 'Software Developer', description: 'Develops applications' })
+await db.Industry.create({ name: 'Technology', naicsCode: '5112' })
+
+// AI generation is grounded against real reference data
+const icp = await db.ICP.create({
+  asHint: 'Engineers who build software',  // Matches "Software Developer"
+  atHint: 'Tech companies',                 // Matches "Technology"
 })
 
-// One call generates Lead + Company + relationships
-const lead = await db.Lead.create({ name: 'Acme Corp' }, { cascade: true })
-const company = await lead.company  // Already exists, fully typed
+const occupation = await icp.as
+// => { title: 'Software Developer', ... } — matched via semantic search, not hallucinated
 ```
 
 ---
 
-## The Problem
+## The Core Insight
 
-**Before ai-database:** AI generates orphaned data
+Traditional databases require foreign keys at schema time. When generating with AI, this fragments context: the model must juggle content creation and referential integrity simultaneously.
 
-```typescript
-// Generate a lead...
-const lead = await ai`generate a sales lead`
-
-// Now what?
-// - Where do you store it?
-// - How do you link it to a company?
-// - What if the company already exists?
-// - How do you query related data?
-
-// You end up with:
-const leadId = await db.insert('leads', lead)
-const companyId = await db.insert('companies', { name: lead.company })
-await db.insert('lead_companies', { leadId, companyId })
-// Manual. Fragile. No type safety. N+1 queries everywhere.
-```
+ai-database inverts this paradigm. **Relationship operators become workflow instructions**, not schema constraints:
 
-**After ai-database:** AI respects your schema
+1. **Generate** the entity with full semantic context intact
+2. **Link** as a post-processing step via insertion or vector search
 
-```typescript
-const { db } = DB({
-  Lead: {
-    name: 'string',
-    company: 'Target company ~>Company',  // Fuzzy match existing or generate
-    score: 'number',
-  },
-  Company: { name: 'string', industry: 'string' }
-})
+This separation eliminates context fragmentation during generation and produces human-readable relationship labels ("Software Developers") instead of opaque IDs (`occ_1547`).
 
-// One line. AI finds existing company or creates one.
-const lead = await db.Lead.create({ name: 'John', companyHint: 'enterprise tech' })
+---
 
-// Relationships just work. Batch loaded. Type safe.
-const company = await lead.company
-```
+## The Four Operators
 
----
+ai-database provides four relationship operators that control how entities connect. They combine two dimensions:
+
+| | **Create New** | **Search Existing** |
+|---|---|---|
+| **Link TO target** | `->` Forward Exact | `~>` Forward Fuzzy |
+| **Link FROM target** | `<-` Backward Exact | `<~` Backward Fuzzy |
+
+### Quick Reference
+
+| Operator | Direction | Match Mode | When to Use |
+|----------|-----------|------------|-------------|
+| `->` | forward | exact | Creating child entities (Blog → Posts) |
+| `~>` | forward | fuzzy | Reusing existing entities (Campaign → Audience) |
+| `<-` | backward | exact | Aggregation queries (Blog collects Posts) |
+| `<~` | backward | fuzzy | Grounding against reference data (ICP → Occupation) |
 
-## Why ai-database?
+### Understanding the Operators
 
-| Pain | Solution |
-|------|----------|
-| N+1 queries loading relationships | **Promise pipelining** - chain without await, batch automatically |
-| AI data disconnected from schema | **Relationship operators** (`->`, `~>`, `<-`, `<~`) for AI-native linking |
-| No types for AI-generated data | **Type-safe schema inference** - full TypeScript support |
-| Manual relationship management | **Cascade generation** - create entity graphs in one call |
+**Direction** determines who owns the relationship:
+- **Forward** (`->`, `~>`): Current entity links TO the target
+- **Backward** (`<-`, `<~`): Target entity links FROM the current entity
+
+**Match Mode** determines how the target is resolved:
+- **Exact** (`->`, `<-`): Create a new entity, then link to it
+- **Fuzzy** (`~>`, `<~`): Search existing entities via semantic similarity
 
 ---
 
-## Quick Start
+## Example 1: Grounding Against Reference Data (`<~`)
 
-```typescript
-import { DB } from 'ai-database'
+The backward fuzzy operator grounds AI-generated content against authoritative reference data. This is the **semantic grounding** pattern.
 
+```typescript
 const { db } = DB({
-  Lead: { name: 'string', company: 'Company.leads' },
-  Company: { name: 'string' }
+  // Generative entity that grounds against reference data
+  IdealCustomerProfile: {
+    as: 'Who are they? (e.g. "Developers") <~Occupation',
+    at: 'Where do they work? (e.g. "FinTech startups") <~Industry',
+    are: 'What are they doing? (e.g. "building APIs") <~Task',
+    using: 'What are they using? (e.g. "Node.js") <~Tool',
+    to: 'What is their goal? (e.g. "ship faster") <~Outcome',
+  },
+
+  // Reference data seeded from O*NET, NAICS, etc.
+  Occupation: {
+    $seed: 'https://onet.data/occupations.tsv',
+    $id: '$.oNETSOCCode',
+    title: '$.title',
+    description: '$.description',
+  },
+  Industry: {
+    $seed: 'https://naics.data/industries.tsv',
+    $id: '$.naicsCode',
+    name: '$.title',
+  },
+  Task: { name: 'string' },
+  Tool: { name: 'string' },
+  Outcome: { description: 'string' },
 })
+```
 
-// Chain without await
-const leads = db.Lead.list()
-const qualified = await leads.filter(l => l.score > 80)
+**How it works:**
 
-// Batch relationship loading
-const enriched = await leads.map(lead => ({
-  name: lead.name,
-  company: lead.company,  // Batch loaded!
-}))
+1. AI generates ICP with `as: "Engineers who build software"`
+2. Runtime embeds the text and searches the `Occupation` collection
+3. Best match found: "Software Developer" (via vector similarity)
+4. Link created with human-readable label: `"Software Developer"`
+
+**Key behaviors:**
+- Uses embedding similarity to find the best match
+- Returns `null` if no semantic match found (doesn't hallucinate)
+- Grounds generated content against curated reference data
+- Perfect for taxonomies, categories, and standardized values
+
+### Union Types for Fallback Search
+
+When multiple collections could contain the best match:
+
+```typescript
+IdealCustomerProfile: {
+  as: '<~Occupation|Role|JobType',      // Search Occupation first, then Role, then JobType
+  using: '<~Tool|Technology|Product',   // Search multiple collections in priority order
+}
 ```
 
 ---
 
-## Promise Pipelining
+## Example 2: Content Generation with Cascade (`->`, `<-`)
 
-Chain database operations without `await`:
+The forward and backward exact operators create hierarchical content. This is the **cascading generation** pattern.
 
 ```typescript
-const leads = db.Lead.list()
-const topLeads = leads.filter(l => l.score > 80)
-const names = topLeads.map(l => l.name)
+const { db } = DB({
+  Blog: {
+    title: 'string',
+    description: 'string',
+    topics: ['List 5 topics covered ->Topic'],  // Creates Topic children
+    posts: ['<-Post'],                           // Aggregates Post children
+  },
+  Topic: {
+    name: 'string',
+    titles: ['List 3 blog post titles ->Post'], // Creates Post children
+  },
+  Post: {
+    title: 'string',
+    synopsis: 'string',
+    content: 'markdown',
+    blog: '->Blog',     // Links back to parent Blog
+    topic: '->Topic',   // Links to Topic
+  },
+})
 
-// Only await when you need the result
-const result = await names
+// One call generates the entire blog structure
+const blog = await db.Blog.create(
+  { title: 'AI Engineering', description: 'Building with LLMs' },
+  { cascade: true, maxDepth: 3 }
+)
+
+// Topics were auto-generated
+const topics = await blog.topics
+// => [{ name: 'Prompt Engineering' }, { name: 'RAG Systems' }, ...]
+
+// Posts were auto-generated under each topic
+const posts = await topics[0].titles
+// => [{ title: 'Getting Started with Prompts' }, ...]
+
+// Backward refs enable aggregation queries
+const allPosts = await blog.posts
+// => All posts that reference this blog
 ```
 
-## Batch Relationship Loading
+### Forward Exact (`->`)
 
-Eliminate N+1 queries automatically:
+Creates child entities that belong to the parent:
 
 ```typescript
-// Old way - N+1 queries
-const leads = await db.Lead.list()
-for (const lead of leads) {
-  const company = await db.Company.get(lead.companyId)  // N queries!
+Startup: {
+  founders: ['Who are the founders? ->Founder'],   // Creates Founder entities
+  businessModel: 'What is the business model? ->LeanCanvas',
 }
+```
 
-// New way - batch loaded
-const enriched = await db.Lead.list().map(lead => ({
-  lead,
-  company: lead.company,  // All companies loaded in ONE query
-}))
+**Key behaviors:**
+- Text before `->` is the AI generation prompt
+- If a value is provided, uses it instead of generating
+- Optional fields (`->Type?`) skip generation when not provided
+- Nested forward fields cascade automatically
+
+### Backward Exact (`<-`)
+
+Creates inverse relationships for aggregation:
+
+```typescript
+Blog: {
+  posts: ['<-Post'],        // All posts that reference this blog
+},
+Post: {
+  blog: '->Blog',           // Forward reference to parent
+}
 ```
 
-## Natural Language Queries
+**Key behaviors:**
+- Creates inverted edge direction (Post → Blog)
+- Enables reverse lookups and aggregation queries
+- Works with explicit backrefs: `['<-Post.blog']`
+- Handles self-referential trees: `children: ['<-Node.parent']`
+
+### Forward Fuzzy (`~>`)
 
-Ask your database questions:
+Searches existing entities first, creates if not found:
 
 ```typescript
-const results = await db.Lead`who closed deals this month?`
-const pending = await db.Order`what's stuck in processing?`
+Campaign: {
+  audience: 'Target audience ~>Audience',  // Find existing or create new
+}
+
+// If "Enterprise" audience exists, reuses it
+const campaign = await db.Campaign.create({
+  audienceHint: 'Big companies with 1000+ employees'
+})
+const audience = await campaign.audience
+// => { name: 'Enterprise', ... } — reused existing!
 ```
 
+**Key behaviors:**
+- Searches via semantic similarity using `${fieldName}Hint`
+- Reuses existing entity if match exceeds threshold
+- Generates new entity if no match found
+- Generated entities marked with `$generated: true`
+
 ---
 
-## Real-World Examples
+## Example 3: Startup Generator (Mixed Operators)
 
-### Sales Pipeline
+A complete example showing all four operators working together:
 
 ```typescript
 const { db } = DB({
-  Lead: {
+  Startup: {
+    $instructions: 'Generate a B2B SaaS startup',
     name: 'string',
-    email: 'string',
-    score: 'number',
-    company: 'Company.leads',
+    idea: 'What problem does this solve? <-Idea',           // Idea spawns Startup
+    founders: ['Who are the founding team? ->Founder'],      // Create founders
+    customer: 'Who is the target customer? ~>CustomerPersona', // Find existing
+    industry: 'What industry? <~Industry',                   // Ground to NAICS
   },
-  Company: {
-    name: 'string',
-    industry: 'string',
-  }
+  Idea: { problem: 'string', solution: 'string' },
+  Founder: { name: 'string', role: 'string' },
+  CustomerPersona: { title: 'string', painPoints: 'string' },
+  Industry: { name: 'string', naicsCode: 'string' },
+})
+
+// Pre-populate reference data
+await db.Industry.create({ name: 'Technology', naicsCode: '5112' })
+await db.CustomerPersona.create({
+  title: 'VP of Engineering',
+  painPoints: 'Managing distributed teams',
 })
 
-// Find high-value leads with their companies
-const qualified = await db.Lead.list()
-  .filter(lead => lead.score > 80)
-  .map(lead => ({
-    lead,
-    company: lead.company,
-  }))
+// Generate complete startup with grounded relationships
+const startup = await db.Startup.create(
+  { name: 'DevFlow' },
+  { cascade: true, maxDepth: 2 }
+)
+
+// Relationships resolved appropriately:
+const idea = await startup.idea        // Created new (->)
+const founders = await startup.founders // Created new ([->])
+const customer = await startup.customer // Matched existing (~>)
+const industry = await startup.industry // Grounded to reference (<~)
+```
+
+---
+
+## Threshold Syntax
+
+For fuzzy operators (`~>` and `<~`), configure the similarity threshold:
+
+### Field-Level Thresholds
 
-// Ask questions naturally
-const results = await db.Lead`who hasn't responded in 2 weeks?`
+```typescript
+Event: {
+  venue: 'Where is the event? ~>Venue(0.9)',     // High threshold - strict match
+  sponsor: 'Event sponsor ~>Company(0.5)',       // Low threshold - lenient match
+}
 ```
 
-### Customer Success
+### Entity-Level Thresholds
 
 ```typescript
-const { db } = DB({
-  Customer: {
-    name: 'string',
-    healthScore: 'number',
-    mrr: 'number',
-    csm: 'User.customers',
-  },
-  User: { name: 'string' }
-})
+Startup: {
+  $fuzzyThreshold: 0.85,  // Apply to all ~> and <~ fields
+  customer: '~>Customer',
+  competitor: '~>Company',
+}
+```
+
+**Threshold values:**
+- `0.9` - Very strict: Only near-exact semantic matches
+- `0.7` - Default: Balanced matching
+- `0.5` - Lenient: Accept loosely related matches
+
+---
+
+## Cascade Generation
+
+Build complex entity graphs from a single `create()` call:
+
+```typescript
+const company = await db.Company.create(
+  { name: 'TechCorp' },
+  {
+    cascade: true,
+    maxDepth: 4,
+    onProgress: (p) => console.log(`${p.totalEntitiesCreated} created`),
+  }
+)
 
-// At-risk customers with their CSMs
-const atRisk = await db.Customer.list()
-  .filter(c => c.healthScore < 50)
-  .map(c => ({
-    customer: c,
-    csm: c.csm,
-    mrr: c.mrr,
-  }))
+// Entire org chart generated: Company → Departments → Teams → Employees
 ```
 
-### Order Management
+### Cascade Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `cascade` | `boolean` | `false` | Enable cascade generation |
+| `maxDepth` | `number` | `0` | Maximum recursion depth |
+| `cascadeTypes` | `string[]` | - | Only cascade to these types |
+| `onProgress` | `function` | - | Progress callback |
+| `onError` | `function` | - | Error handler |
+| `stopOnError` | `boolean` | `false` | Stop on first error |
+
+---
+
+## Special Variables
+
+### `$instructions`
+
+Entity-level prompting that guides AI generation:
 
 ```typescript
-const { db } = DB({
-  Order: {
-    status: 'string',
-    total: 'number',
-    customer: 'Customer.orders',
-    items: ['OrderItem.order'],
-  },
-  OrderItem: { product: 'string', quantity: 'number' },
-  Customer: { name: 'string' }
-})
+Character: {
+  $instructions: 'This character is from a medieval fantasy setting',
+  name: 'string',
+  backstory: 'What is their history?',  // Influenced by $instructions
+}
+```
+
+Template variables resolve against entity data:
+
+```typescript
+Problem: {
+  $instructions: `
+    Identify problems for occupation: {task.occupation.title}
+    in industry: {task.occupation.industry.name}
+  `,
+  task: '<-Task',
+  description: 'string',
+}
+```
+
+### `$context`
 
-// Pending orders with all details
-const pending = await db.Order
-  .find({ status: 'pending' })
-  .map(order => ({
-    order,
-    customer: order.customer,
-    items: order.items,
-  }))
+Explicit context dependencies pre-fetched before generation:
+
+```typescript
+Ad: {
+  $context: ['Startup', 'ICP'],
+  $instructions: 'Generate ad for {startup.name} targeting {icp.as}',
+  headline: 'string (30 chars)',
+}
 ```
 
 ---
@@ -223,11 +393,11 @@ const pending = await db.Order
 Define once, get typed operations everywhere:
 
 ```typescript
-const { db, events, actions, nouns, verbs } = DB({
+const { db, events, actions } = DB({
   Post: {
     title: 'string',
     content: 'markdown',
-    author: 'Author.posts',  // Creates both directions
+    author: 'Author.posts',  // Creates bidirectional relationship
   },
   Author: {
     name: 'string',
@@ -261,21 +431,48 @@ Post: { tags: ['Tag.posts'] }
 
 ---
 
-## CRUD Operations
+## Promise Pipelining
+
+Chain database operations without `await`:
+
+```typescript
+const leads = db.Lead.list()
+const topLeads = leads.filter(l => l.score > 80)
+const names = topLeads.map(l => l.name)
+
+// Only await when you need the result
+const result = await names
+```
+
+### Batch Relationship Loading
+
+Eliminate N+1 queries automatically:
+
+```typescript
+// All companies loaded in ONE query
+const enriched = await db.Lead.list().map(lead => ({
+  lead,
+  company: lead.company,
+}))
+```
+
+---
 
-All operations return `DBPromise` for chaining:
+## CRUD Operations
 
 ```typescript
 // Read
 const lead = await db.Lead.get('lead-123')
 const leads = await db.Lead.list()
-const first = await db.Lead.first()
 const found = await db.Lead.find({ status: 'active' })
 
 // Search
 const results = await db.Lead.search('enterprise SaaS')
 
-// Write (returns regular Promise)
+// Natural language queries
+const pending = await db.Order`what's stuck in processing?`
+
+// Write
 const lead = await db.Lead.create({ name: 'Acme Corp' })
 await db.Lead.update(lead.$id, { score: 90 })
 await db.Lead.delete(lead.$id)
@@ -307,36 +504,23 @@ events.on('*.updated', event => {
 })
 ```
 
+---
+
 ## forEach - Large-Scale Processing
 
-Process thousands of items with concurrency, progress tracking, and error handling:
+Process thousands of items with concurrency and error handling:
 
 ```typescript
-// Simple iteration
-await db.Lead.forEach(lead => {
-  console.log(lead.name)
-})
-
-// With AI and concurrency
 const result = await db.Lead.forEach(async lead => {
   const analysis = await ai`analyze ${lead}`
   await db.Lead.update(lead.$id, { analysis })
 }, {
   concurrency: 10,
-  onProgress: p => console.log(`${p.completed}/${p.total} (${p.rate.toFixed(1)}/s)`),
-})
-
-// With error handling and retries
-await db.Order.forEach(async order => {
-  await sendInvoice(order)
-}, {
-  concurrency: 5,
   maxRetries: 3,
-  retryDelay: attempt => 1000 * Math.pow(2, attempt),  // Exponential backoff
-  onError: (err, order) => err.code === 'RATE_LIMIT' ? 'retry' : 'continue',
+  retryDelay: attempt => 1000 * Math.pow(2, attempt),
+  onProgress: p => console.log(`${p.completed}/${p.total}`),
+  onError: (err, lead) => err.code === 'RATE_LIMIT' ? 'retry' : 'continue',
 })
-
-console.log(`Completed: ${result.completed}, Failed: ${result.failed}`)
 ```
 
 ### forEach Options
@@ -347,9 +531,9 @@ console.log(`Completed: ${result.completed}, Failed: ${result.failed}`)
 | `maxRetries` | `number` | Retries per item (default: 0) |
 | `retryDelay` | `number \| fn` | Delay between retries |
 | `onProgress` | `fn` | Progress callback |
-| `onError` | `'continue' \| 'retry' \| 'skip' \| 'stop' \| fn` | Error handling |
+| `onError` | `fn` | Error handling |
 | `timeout` | `number` | Timeout per item in ms |
-| `persist` | `boolean \| string` | Enable durability (string = custom action name) |
+| `persist` | `boolean \| string` | Enable durability |
 | `resume` | `string` | Resume from action ID |
 
 ### Durable forEach
@@ -357,28 +541,14 @@ console.log(`Completed: ${result.completed}, Failed: ${result.failed}`)
 Persist progress to survive crashes:
 
 ```typescript
-// Enable persistence - auto-names action as "Lead.forEach"
 const result = await db.Lead.forEach(processLead, {
   concurrency: 10,
-  persist: true,
-})
-
-console.log(`Action ID: ${result.actionId}`)
-```
-
-Custom action name:
-
-```typescript
-await db.Lead.forEach(processLead, {
-  persist: 'analyze-leads',  // Custom action name
+  persist: 'analyze-leads',
 })
-```
-
-Resume after a crash:
 
-```typescript
+// Resume after crash
 await db.Lead.forEach(processLead, {
-  resume: 'action-123',  // Skips already-processed items
+  resume: result.actionId,
 })
 ```
 
@@ -415,673 +585,714 @@ DATABASE_URL=sqlite://./data   # SQLite
 DATABASE_URL=:memory:          # in-memory
 ```
 
-## Documentation
+---
+
+## Cloudflare Workers Deployment
 
-- [Full Documentation](https://primitives.org.ai/database)
-- [CRUD Operations](https://primitives.org.ai/database/create)
-- [Schema Types](https://primitives.org.ai/database/schema)
-- [Events](https://primitives.org.ai/database/events)
+ai-database provides dedicated exports for Cloudflare Workers deployment and RPC client consumption.
 
-## Document Database Interface
+### /worker Export
 
-In addition to the schema-first graph model, `ai-database` also exports environment-agnostic types for document-based storage (MDX files with frontmatter). These types are used by `@mdxdb/*` adapters and work in any JavaScript runtime (Node.js, Bun, Deno, Workers, Browser).
+Use the `/worker` export when deploying ai-database as a Cloudflare Worker service:
 
 ```typescript
-import type {
-  DocumentDatabase,
-  DocListOptions,
-  DocSearchOptions,
-  Document,
-} from 'ai-database'
+// worker.ts - the ai-database service
+import { DatabaseWorker, DatabaseDO } from 'ai-database/worker'
 
-// The DocumentDatabase interface
-interface DocumentDatabase<TData> {
-  list(options?: DocListOptions): Promise<DocListResult<TData>>
-  search(options: DocSearchOptions): Promise<DocSearchResult<TData>>
-  get(id: string, options?: DocGetOptions): Promise<Document<TData> | null>
-  set(id: string, doc: Document<TData>, options?: DocSetOptions): Promise<DocSetResult>
-  delete(id: string, options?: DocDeleteOptions): Promise<DocDeleteResult>
-  close?(): Promise<void>
-}
+export { DatabaseDO }
+export default DatabaseWorker
 ```
 
-### Document Types
+```jsonc
+// wrangler.jsonc
+{
+  "name": "ai-database",
+  "main": "src/worker.ts",
+  "compatibility_date": "2024-01-01",
+  "durable_objects": {
+    "bindings": [
+      { "name": "DATABASE_DO", "class_name": "DatabaseDO" }
+    ]
+  }
+}
+```
 
-| Type | Description |
-|------|-------------|
-| `Document<TData>` | MDX document with id, type, context, data, and content |
-| `DocumentDatabase<TData>` | Interface for document storage adapters |
-| `DocListOptions` | Options for listing documents (limit, offset, sortBy, type, prefix) |
-| `DocListResult<TData>` | List result with documents, total, hasMore |
-| `DocSearchOptions` | Search options (query, fields, semantic) |
-| `DocSearchResult<TData>` | Search result with scores |
-| `DocGetOptions` | Get options (includeAst, includeCode) |
-| `DocSetOptions` | Set options (createOnly, updateOnly, version) |
-| `DocSetResult` | Set result (id, version, created) |
-| `DocDeleteOptions` | Delete options (soft, version) |
-| `DocDeleteResult` | Delete result (id, deleted) |
-
-### View Types
-
-For bi-directional relationship rendering:
+### /client Export
 
-| Type | Description |
-|------|-------------|
-| `ViewManager` | Interface for managing views |
-| `ViewDocument` | View template definition |
-| `ViewContext` | Context for rendering a view |
-| `ViewRenderResult` | Rendered markdown and entities |
-| `ViewSyncResult` | Mutations from extracting edited markdown |
-| `DocumentDatabaseWithViews` | Database with view support |
+Use the `/client` export when consuming ai-database from another worker or HTTP client:
 
-### Usage with @mdxdb adapters
+**With Cloudflare Service Bindings (RPC):**
 
 ```typescript
-// Filesystem adapter
-import { createFsDatabase } from '@mdxdb/fs'
-const db = createFsDatabase({ root: './content' })
+// consumer-worker.ts
+import type { DatabaseService } from 'ai-database/worker'
 
-// API adapter
-import { createApiDatabase } from '@mdxdb/api'
-const db = createApiDatabase({ baseUrl: 'https://api.example.com' })
+interface Env {
+  AI_DATABASE: Service<DatabaseService>
+}
 
-// SQLite adapter
-import { createSqliteDatabase } from '@mdxdb/sqlite'
-const db = createSqliteDatabase({ path: './data.db' })
+export default {
+  async fetch(request: Request, env: Env) {
+    // Direct RPC via service binding - no HTTP overhead
+    const service = env.AI_DATABASE.connect('my-namespace')
+    const post = await service.create('Post', { title: 'Hello' })
+    return Response.json(post)
+  }
+}
+```
 
-// Same DocumentDatabase interface regardless of backend
-const doc = await db.get('posts/hello-world')
-await db.set('posts/new', { data: { title: 'New Post' }, content: '# Hello' })
+```jsonc
+// consumer wrangler.jsonc
+{
+  "services": [
+    { "binding": "AI_DATABASE", "service": "ai-database" }
+  ]
+}
 ```
 
----
+**With HTTP Client (rpc.do):**
 
-## Relationship Operators
+```typescript
+import { createDatabaseClient, DB } from 'ai-database/client'
 
-ai-database provides four relationship operators that control how entities are linked and how AI generation flows through your schema. These operators combine two dimensions:
+// Connect to production
+const client = createDatabaseClient('https://ai-database.workers.dev')
+const service = client.connect('my-namespace')
 
-- **Direction**: Forward (`->`, `~>`) vs Backward (`<-`, `<~`)
-- **Match Mode**: Exact (strict foreign key) vs Fuzzy (semantic/AI-driven matching)
+// CRUD operations
+const post = await service.create('Post', { title: 'Hello', content: 'World' })
+const posts = await service.list('Post', { limit: 10 })
+const found = await service.get('Post', post.$id)
 
-### Operator Reference
+// Search
+const results = await service.search('Post', 'hello')
+const semantic = await service.semanticSearch('Post', 'greeting posts')
 
-| Operator | Direction | Match Mode | Description |
-|----------|-----------|------------|-------------|
-| `->` | forward | exact | Creates and links to a new entity (strict FK) |
-| `~>` | forward | fuzzy | Searches existing entities first, generates if no match |
-| `<-` | backward | exact | References an existing entity by ID |
-| `<~` | backward | fuzzy | Finds existing entities via semantic search |
+// Relationships
+await service.relate('Post', post.$id, 'author', 'User', userId)
+const authors = await service.related('Post', post.$id, 'author')
 
-### 1. Forward Exact (`->`)
+// Events
+await service.emit({ event: 'Post.published', actor: userId, object: post.$id })
+const events = await service.listEvents({ event: 'Post.published' })
+```
+
+### TypeScript Setup for Service Bindings
 
-The forward exact operator creates one-to-one or one-to-many relationships where the target entity is auto-generated if not provided.
+For proper type inference with service bindings, import the worker types:
 
 ```typescript
-const { db } = DB({
-  Startup: {
-    name: 'string',
-    idea: 'What is the core idea? ->Idea',           // One-to-one, auto-generated
-    founders: ['Who are the founders? ->Founder'],   // One-to-many, auto-generated
-  },
-  Idea: { description: 'string', solution: 'string' },
-  Founder: { name: 'string', role: 'string' },
-})
+// types.ts
+import type { DatabaseService } from 'ai-database/worker'
 
-// Creating a Startup auto-generates the Idea and Founders
-const startup = await db.Startup.create({ name: 'Acme' })
+export interface Env {
+  AI_DATABASE: Service<DatabaseService>
+  // ... other bindings
+}
+```
 
-const idea = await startup.idea
-// => { $id: '...', $type: 'Idea', description: '...', solution: '...' }
+---
 
-const founders = await startup.founders
-// => [{ $id: '...', $type: 'Founder', name: '...', role: '...' }, ...]
-```
+## Common Patterns
 
-**Key behaviors:**
-- Text before `->` is used as the AI generation prompt
-- If a value is provided, it's used instead of generating new
-- Optional fields (`->Type?`) skip generation when not provided
-- Nested forward exact fields cascade automatically
+### Self-Referential Trees
 
 ```typescript
-// Skip generation by providing an ID
-const existingIdea = await db.Idea.create({ description: 'My idea' })
-const startup = await db.Startup.create({
-  name: 'Acme',
-  idea: existingIdea.$id  // Uses existing, doesn't generate
-})
+Node: {
+  value: 'string',
+  parent: '->Node?',
+  children: ['<-Node.parent'],
+}
 ```
 
-### 2. Forward Fuzzy (`~>`)
-
-The forward fuzzy operator first searches for semantically similar existing entities. If a match is found above the similarity threshold, it reuses that entity. Otherwise, it generates a new one.
+### Union Types for Polymorphic References
 
 ```typescript
-const { db } = DB({
-  Campaign: {
-    name: 'string',
-    audience: 'Target audience for campaign ~>Audience',
-  },
-  Audience: { name: 'string', description: 'string' },
-})
+Comment: {
+  content: 'string',
+  target: '->Post|Article|Video',
+}
 
-// Create some audiences first
-await db.Audience.create({
-  name: 'Enterprise',
-  description: 'Large corporations with 1000+ employees'
-})
-await db.Audience.create({
-  name: 'SMB',
-  description: 'Small businesses with less than 50 employees'
-})
+const target = await comment.target
+console.log(target.$matchedType)  // 'Post', 'Article', or 'Video'
+```
 
-// This will find the Enterprise audience via semantic match
-const campaign = await db.Campaign.create({
-  name: 'Enterprise Sales Push',
-  audienceHint: 'Big companies with thousands of employees'
-})
+### Symmetric Relationships
 
-const audience = await campaign.audience
-// => { $id: '...', name: 'Enterprise', ... } (reused existing!)
+```typescript
+Team: {
+  name: 'string',
+  members: ['->Member'],
+},
+Member: {
+  name: 'string',
+  team: '<-Team',
+}
 ```
 
-**Key behaviors:**
-- Searches existing entities of the target type via semantic similarity
-- `${fieldName}Hint` provides context for matching (e.g., `audienceHint`)
-- If no match exceeds threshold, generates a new entity
-- Generated entities are marked with `$generated: true`
+---
 
-### 3. Backward Exact (`<-`)
+## Document Database Interface
 
-The backward exact operator creates inverse relationships, enabling aggregation queries. The edge direction is inverted - child entities point TO the parent.
+In addition to the schema-first graph model, `ai-database` exports environment-agnostic types for document-based storage (MDX files with frontmatter):
 
 ```typescript
-const { db } = DB({
-  Blog: {
-    name: 'string',
-    posts: ['<-Post'],  // All posts that reference this blog
-  },
-  Post: {
-    title: 'string',
-    blog: '->Blog',     // Forward reference to parent
-  },
-})
+import type {
+  DocumentDatabase,
+  Document,
+  DocListOptions,
+  DocSearchOptions,
+} from 'ai-database'
+
+// Same interface regardless of backend
+const doc = await db.get('posts/hello-world')
+await db.set('posts/new', { data: { title: 'New Post' }, content: '# Hello' })
+```
 
-// Create the blog, then posts that reference it
-const blog = await db.Blog.create({ name: 'Tech Blog' })
-await db.Post.create({ title: 'Hello World', blog: blog.$id })
-await db.Post.create({ title: 'AI Guide', blog: blog.$id })
+### Usage with @mdxdb adapters
 
-// Backward ref enables aggregation queries
-const blogPosts = await blog.posts
-// => [{ title: 'Hello World', ... }, { title: 'AI Guide', ... }]
+```typescript
+import { createFsDatabase } from '@mdxdb/fs'
+import { createSqliteDatabase } from '@mdxdb/sqlite'
+import { createApiDatabase } from '@mdxdb/api'
+
+const db = createFsDatabase({ root: './content' })
+const db = createSqliteDatabase({ path: './data.db' })
+const db = createApiDatabase({ baseUrl: 'https://api.example.com' })
 ```
 
-**Key behaviors:**
-- Creates inverted edge direction (Post -> Blog, not Blog -> Post)
-- Enables reverse lookups and aggregation queries
-- Works with explicit backrefs: `['<-Post.blog']`
-- Handles self-referential relationships: `children: ['<-Node.parent']`
+---
 
-### 4. Backward Fuzzy (`<~`)
+## Provider Capabilities
 
-The backward fuzzy operator combines semantic matching with inverted edge direction. Perfect for grounding generated content against reference data.
+Different database providers support different features. Use `detectCapabilities()` to check what's available at runtime:
 
 ```typescript
-const { db } = DB({
-  ICP: {
-    as: 'Who are they? <~Occupation',      // Ground against occupations
-    at: 'Where do they work? <~Industry',  // Ground against industries
-  },
-  Occupation: { title: 'string', description: 'string' },
-  Industry: { name: 'string', naicsCode: 'string' },
-})
+import { detectCapabilities, requireCapability, CapabilityNotSupportedError } from 'ai-database'
 
-// Create reference data
-await db.Occupation.create({ title: 'Software Developer', description: 'Writes code' })
-await db.Industry.create({ name: 'Technology', naicsCode: '5112' })
+const capabilities = await detectCapabilities(provider)
 
-// ICP grounds against existing reference data
-const icp = await db.ICP.create({
-  asHint: 'Engineers who build software',
-  atHint: 'Tech companies'
-})
+// Check capabilities
+if (capabilities.hasSemanticSearch) {
+  const results = await provider.semanticSearch('Post', 'machine learning')
+} else {
+  // Fallback to regular search
+  const results = await provider.search('Post', 'machine learning')
+}
 
-const occupation = await icp.as
-// => { title: 'Software Developer', ... } (matched via semantic search)
+// Require capabilities (throws if unavailable)
+requireCapability(capabilities, 'hasEvents')
+provider.on('Post.created', handleCreate)
 ```
 
-**Key behaviors:**
-- Uses AI/embedding similarity to find best match
-- Grounds generated content against curated reference data
-- Returns null if no semantic match found (doesn't generate)
-- Useful for taxonomies, categories, and standardized values
+### Capability Matrix
 
----
+| Capability | MemoryProvider | RDB | DigitalObjects |
+|------------|----------------|-----|----------------|
+| **Semantic Search** | Yes | No | No |
+| **Events API** | Yes | No | No |
+| **Actions API** | Yes | No | No |
+| **Artifacts** | Yes | No | No |
+| **Batch Operations** | Yes | No | No |
 
-## Threshold Syntax
+### Capabilities
 
-For fuzzy operators (`~>` and `<~`), you can configure the similarity threshold that determines when a match is accepted vs when new generation occurs.
+| Capability | Description | Methods Required |
+|------------|-------------|------------------|
+| `hasSemanticSearch` | Vector similarity search | `semanticSearch()`, `setEmbeddingsConfig()` |
+| `hasEvents` | Event emission and subscription | `on()`, `emit()`, `listEvents()` |
+| `hasActions` | Durable action tracking | `createAction()`, `getAction()`, `updateAction()` |
+| `hasArtifacts` | Artifact/cache storage | `getArtifact()`, `setArtifact()` |
+| `hasBatchOperations` | Concurrency-controlled batching | `withConcurrency()` or `mapWithConcurrency()` |
 
-### Field-Level Thresholds
+### Graceful Degradation
 
-Append threshold in parentheses after the type name:
+When a capability isn't available, use fallbacks:
 
 ```typescript
-const { db } = DB({
-  Event: {
-    venue: 'Where is the event? ~>Venue(0.9)',     // High threshold - strict match
-    sponsor: 'Event sponsor ~>Company(0.5)',       // Low threshold - lenient match
-  },
-  Venue: { name: 'string', address: 'string' },
-  Company: { name: 'string' },
-})
+import { detectCapabilities, warnIfUnavailable } from 'ai-database'
+
+const capabilities = await detectCapabilities(provider)
+
+// Log a warning (once) if semantic search unavailable
+warnIfUnavailable(capabilities, 'hasSemanticSearch', 'semanticSearch')
+
+// Use capability with fallback
+async function searchPosts(query: string) {
+  if (capabilities.hasSemanticSearch) {
+    return provider.semanticSearch('Post', query)
+  }
+  return provider.search('Post', query)
+}
 ```
 
-**Threshold values:**
-- `0.9` - Very strict: Only near-exact semantic matches
-- `0.7` - Default: Balanced matching
-- `0.5` - Lenient: Accept loosely related matches
+### Features Requiring Semantic Search
 
-### Entity-Level Thresholds
+When using a provider without semantic search support (e.g., RDB), some features behave differently:
+
+| Feature | With Semantic Search | Without Semantic Search |
+|---------|---------------------|------------------------|
+| `~>` Forward Fuzzy | Matches via vector similarity, falls back to generation | Uses text search fallback, then generates if no match |
+| `<~` Backward Fuzzy | Matches via vector similarity | Uses text search fallback |
+| `db.Entity.semanticSearch()` | Vector similarity search | Throws `CapabilityNotSupportedError` |
+| `db.Entity.hybridSearch()` | Combined FTS + vector search | Throws `CapabilityNotSupportedError` |
+| `db.semanticSearch()` | Global vector search | Throws `CapabilityNotSupportedError` |
 
-Set a default threshold for all fuzzy fields in an entity:
+**Fuzzy Operator Fallback**: When semantic search is unavailable, fuzzy operators (`~>` and `<~`) gracefully degrade to basic text search:
 
 ```typescript
+// Without semantic search, these operators use text matching instead of embeddings
 const { db } = DB({
-  Startup: {
-    $fuzzyThreshold: 0.85,  // Apply to all ~> and <~ fields
-    customer: 'Who is the customer? ~>Customer',
-    competitor: 'Main competitor ~>Company',
+  Article: {
+    category: '~>Category',  // Will use text search fallback
   },
-  Customer: { name: 'string' },
-  Company: { name: 'string' },
+  Category: { name: 'string' }
 })
+
+// Forward fuzzy (~>) tries text search first, generates if no match found
+await db.Article.create({ categoryHint: 'Tech' })  // Searches for 'Tech' in categories
+
+// Backward fuzzy (<~) uses text search only - never generates
+await db.Article.create({ categoryHint: 'Tech' })  // Returns null if no text match
 ```
 
-**Matching behavior:**
-1. If similarity score >= threshold: Reuse existing entity
-2. If similarity score < threshold: Generate new entity (for `~>`) or return null (for `<~`)
+**Explicit Search Methods**: When you need semantic search but it's unavailable, the methods throw with helpful alternatives:
+
+```typescript
+import { CapabilityNotSupportedError, isCapabilityNotSupportedError } from 'ai-database'
+
+try {
+  await db.Post.semanticSearch('machine learning')
+} catch (error) {
+  if (isCapabilityNotSupportedError(error)) {
+    console.log(error.capability)   // 'hasSemanticSearch'
+    console.log(error.alternative)  // 'Use the regular search() method instead...'
+    // Fall back to text search
+    const results = await db.Post.search('machine learning')
+  }
+}
+```
 
 ---
 
-## Cascade Generation
+## Integration with RDB
 
-Cascade generation automatically creates related entities recursively, building complex entity graphs from a single `create()` call.
+[RDB](https://github.com/ai-primitives/rdb) provides a simple relational database backend for ai-database. Use it when you want:
 
-### Basic Cascade
+- Edge-native storage via Cloudflare Durable Objects or D1
+- Simple two-table schema (`_data` and `_rels`)
+- Graph traversal and relationship queries
 
-Enable cascade with the `cascade` option:
+### Creating an RDB Provider Adapter
 
 ```typescript
-const { db } = DB({
-  Company: {
-    name: 'string',
-    departments: ['What departments exist? ->Department'],
-  },
-  Department: {
-    name: 'string',
-    teams: ['What teams work here? ->Team'],
-  },
-  Team: {
-    name: 'string',
-    members: ['Who are the team members? ->Employee'],
-  },
-  Employee: { name: 'string', role: 'string' },
-})
+import { setProvider, DB } from 'ai-database'
+import type { DBProvider, ListOptions, SearchOptions } from 'ai-database'
+import { RDB } from '@dotdo/rdb'
 
-const company = await db.Company.create(
-  { name: 'TechCorp' },
-  { cascade: true, maxDepth: 4 }
-)
+// Adapter to bridge RDB and ai-database interfaces
+class RDBProviderAdapter implements DBProvider {
+  private rdb: RDB
 
-// Entire org chart generated automatically!
-const departments = await company.departments
-const teams = await departments[0].teams
-const members = await teams[0].members
-```
+  constructor(sqlStorage: SqlStorage) {
+    this.rdb = new RDB(sqlStorage)
+  }
 
-### Cascade Options
+  async get(type: string, id: string) {
+    const entity = await this.rdb.get(type, id)
+    if (!entity) return null
+    return { $id: entity.id, $type: entity.type, ...entity }
+  }
 
-| Option | Type | Default | Description |
-|--------|------|---------|-------------|
-| `cascade` | `boolean` | `false` | Enable cascade generation |
-| `maxDepth` | `number` | `0` | Maximum depth of recursive generation |
-| `onProgress` | `function` | - | Callback for progress tracking |
-| `onError` | `function` | - | Error handler callback |
-| `stopOnError` | `boolean` | `false` | Stop cascade on first error |
-| `cascadeTypes` | `string[]` | - | Only cascade to these types |
+  async list(type: string, options?: ListOptions) {
+    const entities = await this.rdb.list(type, options)
+    return entities.map(e => ({ $id: e.id, $type: e.type, ...e }))
+  }
 
-### Depth Control
+  async search(type: string, query: string, options?: SearchOptions) {
+    // RDB uses filter-based search; perform text matching
+    const all = await this.rdb.list(type, options)
+    return all
+      .filter(e => JSON.stringify(e).toLowerCase().includes(query.toLowerCase()))
+      .map(e => ({ $id: e.id, $type: e.type, ...e }))
+  }
 
-Control how deep the cascade goes:
+  async create(type: string, id: string | undefined, data: Record<string, unknown>) {
+    const entity = await this.rdb.create(type, data, id)
+    return { $id: entity.id, $type: entity.type, ...entity }
+  }
 
-```typescript
-// maxDepth: 0 - No cascade (default)
-const root = await db.Root.create({ name: 'Test' }, { cascade: true, maxDepth: 0 })
-// root.items is empty - not generated
+  async update(type: string, id: string, data: Record<string, unknown>) {
+    const entity = await this.rdb.update(type, id, data)
+    return { $id: entity.id, $type: entity.type, ...entity }
+  }
 
-// maxDepth: 1 - Only immediate children
-const parent = await db.Parent.create({ name: 'P' }, { cascade: true, maxDepth: 1 })
-// parent.children generated, but grandchildren are not
+  async delete(type: string, id: string): Promise<boolean> {
+    const exists = await this.rdb.get(type, id)
+    if (!exists) return false
+    await this.rdb.delete(type, id)
+    return true
+  }
 
-// maxDepth: 3 - Three levels deep
-const company = await db.Company.create({ name: 'X' }, { cascade: true, maxDepth: 3 })
-// company -> departments -> teams -> employees (stops at employees)
-```
+  async related(type: string, id: string, relation: string) {
+    const entities = await this.rdb.related(type, id, relation)
+    return entities.map(e => ({ $id: e.id, $type: e.type, ...e }))
+  }
 
-### Progress Tracking
+  async relate(fromType: string, fromId: string, relation: string, toType: string, toId: string, metadata?: object) {
+    await this.rdb.relate(fromType, fromId, relation, toType, toId, metadata)
+  }
 
-Monitor cascade generation progress:
+  async unrelate(fromType: string, fromId: string, relation: string, toType: string, toId: string) {
+    await this.rdb.unrelate(fromType, fromId, relation, toType, toId)
+  }
+}
 
-```typescript
-const company = await db.Company.create(
-  { name: 'TechCorp' },
-  {
-    cascade: true,
-    maxDepth: 4,
-    onProgress: (progress) => {
-      console.log(`Phase: ${progress.phase}`)          // 'generating' or 'complete'
-      console.log(`Depth: ${progress.depth}`)          // Current depth level
-      console.log(`Type: ${progress.currentType}`)     // Type being generated
-      console.log(`Total: ${progress.totalEntitiesCreated}`)
-    },
+// Usage in a Durable Object
+export class MyDO extends DurableObject {
+  constructor(ctx: DurableObjectState, env: Env) {
+    super(ctx, env)
+    setProvider(new RDBProviderAdapter(ctx.storage.sql))
   }
-)
+}
+
+// Now use ai-database schema with RDB backend
+const { db } = DB({
+  Post: { title: 'string', author: '->Author.posts' },
+  Author: { name: 'string' },
+})
+
+const author = await db.Author.create({ name: 'Alice' })
+const post = await db.Post.create({ title: 'Hello', author: author.$id })
 ```
 
-### Selective Cascade
+### Limitations with RDB
 
-Only cascade to specific types:
+When using RDB as a provider:
 
-```typescript
-const company = await db.Company.create(
-  { name: 'TechCorp' },
-  {
-    cascade: true,
-    maxDepth: 3,
-    cascadeTypes: ['Department', 'Team'],  // Skip Employee generation
-  }
-)
-```
+- **No semantic search**: Fuzzy operators (`~>`, `<~`) require vector embeddings. Use exact operators (`->`, `<-`) instead, or use MemoryProvider for semantic matching.
+- **No events/actions API**: RDB focuses on core CRUD and relationships.
+- **Text search only**: The `search()` method performs text matching, not semantic similarity.
 
 ---
 
-## Special Variables
+## AI Integration
 
-### `$instructions`
+ai-database integrates with AI providers for two core capabilities:
+
+1. **Entity Generation** - AI-powered content generation for schema fields using `ai-functions`
+2. **Semantic Search** - Vector embeddings for fuzzy matching (`~>`, `<~` operators) and similarity search
+
+### Supported AI Providers
+
+#### For Entity Generation
 
-Entity-level prompting that guides AI generation for all fields:
+Entity generation uses [ai-functions](https://github.com/ai-primitives/ai-primitives/tree/main/packages/ai-functions) which supports:
+
+| Provider | Models | Configuration |
+|----------|--------|---------------|
+| **Anthropic** | claude-3-5-sonnet, claude-3-opus, claude-3-haiku | `ANTHROPIC_API_KEY` |
+| **OpenAI** | gpt-4o, gpt-4-turbo, gpt-3.5-turbo | `OPENAI_API_KEY` |
+| **Google** | gemini-1.5-pro, gemini-1.5-flash | `GOOGLE_API_KEY` |
+| **Local Models** | Ollama, LM Studio, llama.cpp | `AI_BASE_URL` |
 
 ```typescript
-const { db } = DB({
-  Character: {
-    $instructions: 'This character is from a medieval fantasy setting',
-    name: 'string',
-    backstory: 'What is their history?',  // Influenced by $instructions
-  },
+import { DB, configureAIGeneration } from 'ai-database'
+
+// Configure the AI model for entity generation
+configureAIGeneration({
+  model: 'sonnet',        // Model alias (see ai-functions for full list)
+  enabled: true,          // Enable AI generation (default: true)
+  onGenerate: (details) => {
+    // Track generation calls for monitoring
+    console.log(`Generated ${details.entityType} in ${details.latencyMs}ms`)
+    if (details.error) console.error('Generation failed:', details.error)
+  }
 })
 
-const character = await db.Character.create({ name: 'Sir Aldric' })
-// backstory will reference medieval elements (castles, knights, quests)
+const { db } = DB({
+  BlogPost: {
+    title: 'string',
+    content: 'Write a detailed blog post about this topic',  // AI generates this
+    summary: 'Summarize the content in 2 sentences',
+  }
+})
 ```
 
-**Template variables** resolve against entity data:
+#### For Embeddings/Semantic Search
+
+Embedding generation for semantic search can use any provider that produces vector embeddings:
+
+| Provider | Models | Dimensions | Configuration |
+|----------|--------|------------|---------------|
+| **OpenAI** | text-embedding-3-small | 1536 | `OPENAI_API_KEY` |
+| **OpenAI** | text-embedding-3-large | 3072 | `OPENAI_API_KEY` |
+| **Cohere** | embed-english-v3.0 | 1024 | `COHERE_API_KEY` |
+| **Voyage AI** | voyage-large-2 | 1024-4096 | `VOYAGE_API_KEY` |
+| **Local** | sentence-transformers | 384 | Self-hosted |
 
 ```typescript
-const { db } = DB({
-  Problem: {
-    $instructions: `
-      Identify problems for occupation: {task.occupation.title}
-      in industry: {task.occupation.industry.name}
-    `,
-    task: '<-Task',
-    description: 'string',
-  },
-  Task: { name: 'string', occupation: '->Occupation' },
-  Occupation: { title: 'string', industry: '->Industry' },
-  Industry: { name: 'string' },
+import { createMemoryProvider, setProvider } from 'ai-database'
+
+// Configure embedding dimensions to match your provider
+const provider = createMemoryProvider({
+  embeddingDimensions: 1536,  // Match OpenAI text-embedding-3-small
 })
+setProvider(provider)
 ```
 
-### `$context`
+---
 
-Explicit context dependencies that are pre-fetched before generation:
+### Configuring Embedding Generation
+
+Control which fields are embedded for semantic search:
 
 ```typescript
+import { DB } from 'ai-database'
+
 const { db } = DB({
-  Ad: {
-    $context: ['Startup', 'ICP'],  // Pre-fetch these for template resolution
-    $instructions: 'Generate ad for {startup.name} targeting {icp.as}',
-    startup: '<-Startup',
-    headline: 'string (30 chars)',
+  Article: {
+    title: 'string',
+    content: 'markdown',
+    authorId: 'string',    // Won't be embedded (not text content)
   },
-  Startup: { name: 'string', icp: '->ICP' },
-  ICP: { as: 'string' },
+  InternalNote: {
+    text: 'string',
+  }
+}, {
+  embeddings: {
+    // Specify which fields to embed for Article
+    Article: { fields: ['title', 'content'] },
+
+    // Disable embeddings for InternalNote (won't appear in semantic search)
+    InternalNote: false,
+  }
 })
+```
 
-const icp = await db.ICP.create({ as: 'Software Engineers' })
-const startup = await db.Startup.create({ name: 'CodeHelper', icp: icp.$id })
-const ad = await db.Ad.create({ startup: startup.$id })
+#### Embedding Configuration Options
 
-// headline will mention CodeHelper and Software Engineers
-```
+| Option | Type | Description |
+|--------|------|-------------|
+| `fields` | `string[]` | Fields to include in embedding (default: auto-detect text fields) |
+| `false` | `boolean` | Disable embeddings for this entity type |
+
+#### Auto-Detection
 
-**Why use `$context`?**
-- Explicitly declares what entities are needed for generation
-- Enables efficient pre-fetching of related data
-- Makes template variable resolution predictable
-- Supports multiple levels of relationship traversal
+If no `embeddings` config is provided, ai-database automatically embeds:
+- All `string` fields (except those ending in `Id`, `At`, or starting with `$`/`_`)
+- All `markdown` fields
+- String arrays (concatenated)
 
 ---
 
-## Complete Examples
+### Cost and Token Implications
+
+Understanding token usage is critical for production deployments. Here's what triggers AI API calls:
+
+#### Entity Generation Costs
 
-### Example 1: Startup Generator
+| Operation | AI Calls | When |
+|-----------|----------|------|
+| `create()` with prompt fields | 1 per entity | Fields like `'Write a description'` |
+| `create({ cascade: true })` | 1 per cascaded entity | Each `->` forward relation |
+| `create()` with `~>` fuzzy | 1 embedding + search | If no semantic match found, may generate |
 
-A complete startup pitch generator with cascading entity creation:
+**Example: Cascade Cost Estimation**
 
 ```typescript
 const { db } = DB({
-  Startup: {
-    $instructions: 'Generate a B2B SaaS startup',
-    name: 'string',
-    idea: 'What problem does this solve? ->Idea',
-    founders: ['Who are the founding team? ->Founder'],
-    customer: 'Who is the target customer? ~>CustomerPersona',
-  },
-  Idea: {
-    problem: 'string',
-    solution: 'string',
-    differentiator: 'What makes this unique?',
+  Blog: {
+    title: 'string',
+    topics: ['Generate 5 topics ->Topic'],     // Creates 5 Topic entities
   },
-  Founder: {
+  Topic: {
     name: 'string',
-    role: 'string',
-    background: 'Previous experience',
+    posts: ['Generate 3 posts ->Post'],        // Creates 3 Post entities per Topic
   },
-  CustomerPersona: {
+  Post: {
     title: 'string',
-    painPoints: 'string',
-    budget: 'string',
-  },
-})
-
-// Pre-populate customer personas
-await db.CustomerPersona.create({
-  title: 'VP of Engineering',
-  painPoints: 'Managing distributed teams, code quality',
-  budget: '$50k-100k annually',
+    content: 'Write a 500-word blog post',     // AI generates ~500 words
+  }
 })
 
-// Generate complete startup with one call
-const startup = await db.Startup.create(
-  { name: 'DevFlow' },
-  { cascade: true, maxDepth: 2 }
+// This single call generates:
+// - 1 Blog (1 generation call)
+// - 5 Topics (5 generation calls)
+// - 15 Posts (15 generation calls, each ~500 words)
+// Total: 21 AI generation calls
+const blog = await db.Blog.create(
+  { title: 'My Tech Blog' },
+  { cascade: true, maxDepth: 3 }
 )
-
-// Access generated entities
-const idea = await startup.idea
-const founders = await startup.founders
-const customer = await startup.customer  // May match existing or generate new
 ```
 
-### Example 2: Content Management with Grounding
-
-Grounding generated content against reference taxonomies:
+**Cost Control Strategies:**
 
 ```typescript
-const { db } = DB({
-  Article: {
-    $instructions: 'Write a technical blog post',
-    title: 'string',
-    content: 'markdown',
-    category: 'What category? <~Category',     // Ground against existing
-    tags: ['Relevant tags <~Tag'],             // Multiple semantic matches
-    author: '->Author',                        // Auto-generate author
-  },
-  Category: { name: 'string', description: 'string' },
-  Tag: { name: 'string' },
-  Author: { name: 'string', bio: 'string' },
-})
+// 1. Limit cascade depth to control entity count
+await db.Blog.create(data, { cascade: true, maxDepth: 1 })  // Only creates immediate children
 
-// Set up taxonomy
-await db.Category.create({ name: 'Artificial Intelligence', description: 'ML and AI topics' })
-await db.Category.create({ name: 'Web Development', description: 'Frontend and backend' })
-await db.Tag.create({ name: 'Machine Learning' })
-await db.Tag.create({ name: 'Deep Learning' })
-await db.Tag.create({ name: 'React' })
-
-// Create article - content grounded against existing categories
-const article = await db.Article.create({
-  title: 'Introduction to Neural Networks',
-  categoryHint: 'AI and machine learning topics',
-  tagsHint: ['neural network concepts', 'ML fundamentals'],
+// 2. Filter which types cascade
+await db.Blog.create(data, {
+  cascade: true,
+  cascadeTypes: ['Topic']  // Only cascade to Topic, not Post
 })
 
-const category = await article.category
-// => { name: 'Artificial Intelligence', ... } - matched existing!
+// 3. Track costs with onGenerate callback
+let totalTokens = 0
+configureAIGeneration({
+  model: 'sonnet',
+  onGenerate: (details) => {
+    if (details.result) {
+      // Estimate tokens (actual count depends on provider)
+      const inputTokens = details.prompt.length / 4
+      const outputTokens = JSON.stringify(details.result).length / 4
+      totalTokens += inputTokens + outputTokens
+      console.log(`Running total: ~${totalTokens} tokens`)
+    }
+  }
+})
 
-const tags = await article.tags
-// => [{ name: 'Machine Learning' }, { name: 'Deep Learning' }] - matched existing!
+// 4. Use draftOnly for preview without committing
+const draft = await db.Blog.draft({ title: 'Test' })
+// Review draft before creating
+const entity = await draft.resolve()
 ```
 
-### Example 3: Hierarchical Organization Chart
+#### Embedding Costs
+
+| Operation | Embedding Calls | Notes |
+|-----------|-----------------|-------|
+| `create()` | 1 per entity | Embeds text fields on creation |
+| `update()` | 1 if text changed | Re-embeds when text fields update |
+| `semanticSearch()` | 1 for query | Embeds query string |
+| `hybridSearch()` | 1 for query | Embeds query string |
+| `~>` or `<~` resolution | 1 per hint | Embeds the hint text for matching |
+
+---
+
+### Rate Limiting Best Practices
+
+ai-database provides built-in concurrency control to prevent API rate limit errors:
 
-Building a complete org structure with bidirectional navigation:
+#### Using forEach with Concurrency
 
 ```typescript
-const { db } = DB({
-  Company: {
-    name: 'string',
-    ceo: '->Person',
-    departments: ['<-Department'],  // Backward ref for aggregation
-  },
-  Department: {
-    name: 'string',
-    company: '->Company',           // Forward ref to parent
-    head: '->Person',
-    employees: ['<-Person'],        // All people in this department
-  },
-  Person: {
-    name: 'string',
-    role: 'string',
-    department: '->Department?',    // Optional department
-    reportsTo: '->Person?',         // Self-referential hierarchy
-    directReports: ['<-Person.reportsTo'],
+// Process entities with controlled concurrency
+const result = await db.Lead.forEach(async (lead) => {
+  const analysis = await generateAnalysis(lead)
+  await db.Lead.update(lead.$id, { analysis })
+}, {
+  concurrency: 10,         // Max 10 parallel operations
+  maxRetries: 3,           // Retry failed items up to 3 times
+  retryDelay: (attempt) => 1000 * Math.pow(2, attempt),  // Exponential backoff
+
+  // Handle rate limit errors specifically
+  onError: (error, lead) => {
+    if (error.message.includes('rate_limit') || error.message.includes('429')) {
+      return 'retry'       // Retry with exponential backoff
+    }
+    return 'continue'      // Skip this item and continue
   },
-})
-
-// Create company structure
-const company = await db.Company.create({ name: 'TechCorp' })
-const engineering = await db.Department.create({
-  name: 'Engineering',
-  company: company.$id,
-})
 
-const cto = await db.Person.create({
-  name: 'Alice',
-  role: 'CTO',
-  department: engineering.$id,
+  onProgress: (progress) => {
+    console.log(`${progress.completed}/${progress.total} (${progress.failed} failed)`)
+  }
 })
+```
 
-const engineer = await db.Person.create({
-  name: 'Bob',
-  role: 'Senior Engineer',
-  department: engineering.$id,
-  reportsTo: cto.$id,
-})
+#### Provider-Level Concurrency
 
-// Navigate bidirectionally
-const aliceReports = await cto.directReports
-// => [{ name: 'Bob', ... }]
+```typescript
+import { createMemoryProvider } from 'ai-database'
 
-const engineeringTeam = await engineering.employees
-// => [{ name: 'Alice', ... }, { name: 'Bob', ... }]
+// Configure concurrency at the provider level
+const provider = createMemoryProvider({
+  concurrency: 10,  // Global limit on parallel operations
+})
 ```
 
----
+#### Execution Queue for Batch Operations
 
-## Common Patterns
-
-### Union Types for Polymorphic References
+For large-scale operations with different priority levels:
 
 ```typescript
-const { db } = DB({
-  Comment: {
-    content: 'string',
-    target: '->Post|Article|Video',  // Can reference any of these types
-  },
-  Post: { title: 'string' },
-  Article: { title: 'string' },
-  Video: { title: 'string', url: 'url' },
+import { ExecutionQueue } from 'ai-database'
+
+const queue = new ExecutionQueue({
+  concurrency: {
+    priority: 50,    // High-priority operations
+    standard: 20,    // Normal operations
+    flex: 10,        // Low-priority background operations
+    batch: 1000,     // Batch window operations
+  }
 })
 
-const target = await comment.target
-console.log(target.$matchedType)  // 'Post', 'Article', or 'Video'
+// Submit operations with priority
+await queue.submit(
+  () => db.Lead.create({ name: 'Important Lead' }),
+  { priority: 'priority' }  // Runs with higher concurrency
+)
 ```
 
-### Self-Referential Trees
+#### Rate Limit Patterns by Provider
+
+| Provider | Rate Limits | Recommended Concurrency |
+|----------|-------------|------------------------|
+| **OpenAI** | 60-10000 RPM (varies by tier) | 5-50 |
+| **Anthropic** | 60-4000 RPM (varies by tier) | 5-40 |
+| **Cohere** | 100 RPM (trial), 10000 RPM (prod) | 5-100 |
+| **Local (Ollama)** | Limited by hardware | 1-4 |
+
+**Recommended Configuration by Use Case:**
 
 ```typescript
-const { db } = DB({
-  Node: {
-    value: 'string',
-    parent: '->Node?',
-    children: ['<-Node.parent'],
-  },
-})
+// Development/Testing - Low concurrency, fail fast
+configureAIGeneration({ model: 'sonnet', enabled: true })
+await db.Entity.forEach(fn, { concurrency: 2, maxRetries: 1 })
 
-const root = await db.Node.create({ value: 'Root' })
-const child = await db.Node.create({ value: 'Child', parent: root.$id })
+// Production - Moderate concurrency with retries
+await db.Entity.forEach(fn, {
+  concurrency: 10,
+  maxRetries: 3,
+  retryDelay: attempt => 1000 * Math.pow(2, attempt)
+})
 
-const rootChildren = await root.children
-// => [{ value: 'Child', ... }]
+// Batch Processing - Low concurrency, high retry tolerance
+await db.Entity.forEach(fn, {
+  concurrency: 5,
+  maxRetries: 5,
+  retryDelay: attempt => 2000 * Math.pow(2, attempt),
+  timeout: 60000,  // 60 second timeout per item
+  persist: 'batch-job-123',  // Resume on crash
+})
 ```
 
-### Symmetric Relationships
+---
+
+### Disabling AI Generation
+
+For testing or when you want placeholder values instead of AI generation:
 
 ```typescript
-const { db } = DB({
-  Team: {
-    name: 'string',
-    members: ['->Member'],
-  },
-  Member: {
-    name: 'string',
-    team: '<-Team',  // Points back to team
-  },
-})
+import { configureAIGeneration } from 'ai-database'
 
-// Creating team generates members
-const team = await db.Team.create({ name: 'Engineering' }, { cascade: true })
+// Disable AI globally - uses deterministic placeholder values
+configureAIGeneration({ enabled: false })
 
-// Each member can navigate back to team
-const member = (await team.members)[0]
-const memberTeam = await member.team
-// memberTeam.$id === team.$id
+// Or per-DB instance
+const { db } = DB(schema, {
+  aiGeneration: { enabled: false }
+})
 ```
 
+When AI is disabled:
+- Prompt fields generate deterministic placeholder text
+- Fuzzy operators (`~>`, `<~`) fall back to text search
+- No API calls are made to AI providers
+- Tests run faster and don't require API keys
+
 ---
 
 ## Related