ai-database 0.0.0-development → 0.2.0

package/.turbo/turbo-build.log

@@ -0,0 +1,5 @@
+
+
+> ai-database@0.2.0 build /Users/nathanclevenger/projects/mdx.org.ai/primitives/packages/ai-database
+> tsc -p tsconfig.json
+

package/.turbo/turbo-test.log

@@ -0,0 +1,102 @@
+
+> ai-database@0.0.1 test /Users/nathanclevenger/projects/mdx.org.ai/primitives/packages/ai-database
+> vitest
+
+
+ DEV  v2.1.9 /Users/nathanclevenger/projects/mdx.org.ai/primitives/packages/ai-database
+
+ ✓ src/schema.test.ts (83 tests) 13ms
+ ✓ src/memory-provider.test.ts (81 tests) 197ms
+ ❯ src/index.test.ts (47 tests | 1 failed) 13ms
+   × DB integration tests > list and query operations > iterates with options 5ms
+     → expected [] to deeply equal [ 'John' ]
+ ✓ test/edge-cases.test.ts (43 tests) 9ms
+ ✓ test/provider-resolution.test.ts (38 tests) 2ms
+
+⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
+
+ FAIL  src/index.test.ts > DB integration tests > list and query operations > iterates with options
+AssertionError: expected [] to deeply equal [ 'John' ]
+
+- Expected
++ Received
+
+- Array [
+-   "John",
+- ]
++ Array []
+
+ ❯ src/index.test.ts:251:21
+    249|       })
+    250| 
+    251|       expect(names).toEqual(['John'])
+       |                     ^
+    252|     })
+    253|   })
+
+⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯
+
+ Test Files  1 failed | 4 passed (5)
+      Tests  1 failed | 291 passed (292)
+   Start at  14:14:37
+   Duration  673ms (transform 184ms, setup 0ms, collect 232ms, tests 235ms, environment 0ms, prepare 43ms)
+
+ FAIL  Tests failed. Watching for file changes...
+       press h to show help, press q to quit
+c RERUN  src/ai-promise-db.ts 
+
+ ✓ src/index.test.ts (47 tests) 9ms
+ ✓ src/schema.test.ts (83 tests) 7ms
+ ✓ test/edge-cases.test.ts (43 tests) 9ms
+ ✓ test/provider-resolution.test.ts (38 tests) 2ms
+
+ Test Files  4 passed (4)
+      Tests  211 passed (211)
+   Start at  14:35:16
+   Duration  79ms
+
+ PASS  Waiting for file changes...
+       press h to show help, press q to quit
+c RERUN  src/ai-promise-db.ts 
+
+ ✓ src/index.test.ts (47 tests) 10ms
+ ✓ test/edge-cases.test.ts (43 tests) 9ms
+ ✓ src/schema.test.ts (83 tests) 7ms
+ ✓ test/provider-resolution.test.ts (38 tests) 2ms
+
+ Test Files  4 passed (4)
+      Tests  211 passed (211)
+   Start at  14:35:40
+   Duration  70ms
+
+ PASS  Waiting for file changes...
+       press h to show help, press q to quit
+c RERUN  src/ai-promise-db.ts 
+
+ ✓ src/index.test.ts (47 tests) 10ms
+ ✓ test/edge-cases.test.ts (43 tests) 8ms
+ ✓ src/schema.test.ts (83 tests) 6ms
+ ✓ test/provider-resolution.test.ts (38 tests) 2ms
+
+ Test Files  4 passed (4)
+      Tests  211 passed (211)
+   Start at  14:35:53
+   Duration  65ms
+
+ PASS  Waiting for file changes...
+       press h to show help, press q to quit
+c RERUN  src/ai-promise-db.ts 
+
+ ✓ src/index.test.ts (47 tests) 15ms
+ ✓ test/edge-cases.test.ts (43 tests) 11ms
+ ✓ src/schema.test.ts (83 tests) 9ms
+ ✓ test/provider-resolution.test.ts (38 tests) 2ms
+
+ Test Files  4 passed (4)
+      Tests  211 passed (211)
+   Start at  01:20:37
+   Duration  115ms
+
+ PASS  Waiting for file changes...
+       press h to show help, press q to quit
+ ELIFECYCLE  Test failed. See above for more details.

package/README.md

@@ -1,72 +1,427 @@
 # ai-database
 
-[![npm version](https://badge.fury.io/js/ai-database.svg)](https://badge.fury.io/js/ai-database)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+Your data, flowing like conversation.
 
-AI-native database abstraction with hybrid vector search capabilities for synthetic data, tool-calling, and RAG applications.
+```typescript
+import { DB } from 'ai-database'
 
-## Features
-- Hybrid vector search optimized for AI workloads
-- Synthetic data generation and management
-- Tool-calling interface compatible with major AI SDKs
-- Built-in support for RAG (Retrieval Augmented Generation)
-- Seamless integration with mdxdb for document storage
+const { db } = DB({
+  Lead: { name: 'string', company: 'Company.leads' },
+  Company: { name: 'string' }
+})
 
-## Installation
-```bash
-npm install ai-database
-# or
-pnpm add ai-database
-# or
-yarn add ai-database
+// Chain without await
+const leads = db.Lead.list()
+const qualified = await leads.filter(l => l.score > 80)
+
+// Batch relationship loading
+const enriched = await leads.map(lead => ({
+  name: lead.name,
+  company: lead.company,  // Batch loaded!
+}))
 ```
 
-## Quick Start
+## Promise Pipelining
+
+Chain database operations without `await`:
+
 ```typescript
-import { createDatabase } from 'ai-database'
+const leads = db.Lead.list()
+const topLeads = leads.filter(l => l.score > 80)
+const names = topLeads.map(l => l.name)
+
+// Only await when you need the result
+const result = await names
+```
+
+## Batch Relationship Loading
+
+Eliminate N+1 queries automatically:
+
+```typescript
+// Old way - N+1 queries
+const leads = await db.Lead.list()
+for (const lead of leads) {
+  const company = await db.Company.get(lead.companyId)  // N queries!
+}
+
+// New way - batch loaded
+const enriched = await db.Lead.list().map(lead => ({
+  lead,
+  company: lead.company,  // All companies loaded in ONE query
+}))
+```
+
+## Natural Language Queries
 
-// Initialize database with vector search capabilities
-const db = createDatabase({
-  namespace: 'my-app',
-  vectorSearch: true
+Ask your database questions:
+
+```typescript
+const results = await db.Lead`who closed deals this month?`
+const pending = await db.Order`what's stuck in processing?`
+```
+
+---
+
+## Real-World Examples
+
+### Sales Pipeline
+
+```typescript
+const { db } = DB({
+  Lead: {
+    name: 'string',
+    email: 'string',
+    score: 'number',
+    company: 'Company.leads',
+  },
+  Company: {
+    name: 'string',
+    industry: 'string',
+  }
 })
 
-// Store documents with embeddings
-await db.collection('documents').store({
-  content: 'Example document',
-  embeddings: [0.1, 0.2, 0.3]
+// Find high-value leads with their companies
+const qualified = await db.Lead.list()
+  .filter(lead => lead.score > 80)
+  .map(lead => ({
+    lead,
+    company: lead.company,
+  }))
+
+// Ask questions naturally
+const results = await db.Lead`who hasn't responded in 2 weeks?`
+```
+
+### Customer Success
+
+```typescript
+const { db } = DB({
+  Customer: {
+    name: 'string',
+    healthScore: 'number',
+    mrr: 'number',
+    csm: 'User.customers',
+  },
+  User: { name: 'string' }
+})
+
+// At-risk customers with their CSMs
+const atRisk = await db.Customer.list()
+  .filter(c => c.healthScore < 50)
+  .map(c => ({
+    customer: c,
+    csm: c.csm,
+    mrr: c.mrr,
+  }))
+```
+
+### Order Management
+
+```typescript
+const { db } = DB({
+  Order: {
+    status: 'string',
+    total: 'number',
+    customer: 'Customer.orders',
+    items: ['OrderItem.order'],
+  },
+  OrderItem: { product: 'string', quantity: 'number' },
+  Customer: { name: 'string' }
 })
 
-// Perform hybrid search
-const results = await db.collection('documents').search({
-  query: 'example',
-  vector: [0.1, 0.2, 0.3],
-  threshold: 0.8
+// Pending orders with all details
+const pending = await db.Order
+  .find({ status: 'pending' })
+  .map(order => ({
+    order,
+    customer: order.customer,
+    items: order.items,
+  }))
+```
+
+---
+
+## Schema Definition
+
+Define once, get typed operations everywhere:
+
+```typescript
+const { db, events, actions, nouns, verbs } = DB({
+  Post: {
+    title: 'string',
+    content: 'markdown',
+    author: 'Author.posts',  // Creates both directions
+  },
+  Author: {
+    name: 'string',
+    // posts: Post[] auto-created from backref
+  }
 })
 ```
 
-## Tool Integration
-ai-database exports AI-compatible tools that work with any LLM supporting function calling:
+### Field Types
+
+| Type | Description |
+|------|-------------|
+| `string` | Text |
+| `number` | Numeric |
+| `boolean` | True/false |
+| `date` | Date only |
+| `datetime` | Date and time |
+| `markdown` | Rich text |
+| `json` | Structured data |
+| `url` | URL string |
+
+### Relationships
 
 ```typescript
-import { tools } from 'ai-database'
+// One-to-many: Post has one Author, Author has many Posts
+Post: { author: 'Author.posts' }
+
+// Many-to-many: Post has many Tags, Tag has many Posts
+Post: { tags: ['Tag.posts'] }
+```
 
-// Use with any AI SDK (Vercel AI, LangChain, etc)
-const searchTool = tools.vectorSearch({
-  collection: 'documents',
-  namespace: 'my-app'
+---
+
+## CRUD Operations
+
+All operations return `DBPromise` for chaining:
+
+```typescript
+// Read
+const lead = await db.Lead.get('lead-123')
+const leads = await db.Lead.list()
+const first = await db.Lead.first()
+const found = await db.Lead.find({ status: 'active' })
+
+// Search
+const results = await db.Lead.search('enterprise SaaS')
+
+// Write (returns regular Promise)
+const lead = await db.Lead.create({ name: 'Acme Corp' })
+await db.Lead.update(lead.$id, { score: 90 })
+await db.Lead.delete(lead.$id)
+```
+
+### Chainable Methods
+
+```typescript
+db.Lead.list()
+  .filter(l => l.score > 50)
+  .sort((a, b) => b.score - a.score)
+  .limit(10)
+  .map(l => ({ name: l.name, score: l.score }))
+```
+
+---
+
+## Events
+
+React to changes in real-time:
+
+```typescript
+events.on('Lead.created', event => {
+  notifySlack(`New lead: ${event.data.name}`)
+})
+
+events.on('*.updated', event => {
+  logChange(event)
 })
 ```
 
-## Integration with AI Primitives
-ai-database is designed to work seamlessly with other AI Primitives packages:
+## forEach - Large-Scale Processing
 
-- **ai-functions**: Provides database operations as callable AI functions
-- **ai-workflows**: Enables database integration in AI workflow definitions
-- **ai-agents**: Offers database access tools for AI agents
+Process thousands of items with concurrency, progress tracking, and error handling:
+
+```typescript
+// Simple iteration
+await db.Lead.forEach(lead => {
+  console.log(lead.name)
+})
+
+// With AI and concurrency
+const result = await db.Lead.forEach(async lead => {
+  const analysis = await ai`analyze ${lead}`
+  await db.Lead.update(lead.$id, { analysis })
+}, {
+  concurrency: 10,
+  onProgress: p => console.log(`${p.completed}/${p.total} (${p.rate.toFixed(1)}/s)`),
+})
+
+// With error handling and retries
+await db.Order.forEach(async order => {
+  await sendInvoice(order)
+}, {
+  concurrency: 5,
+  maxRetries: 3,
+  retryDelay: attempt => 1000 * Math.pow(2, attempt),  // Exponential backoff
+  onError: (err, order) => err.code === 'RATE_LIMIT' ? 'retry' : 'continue',
+})
+
+console.log(`Completed: ${result.completed}, Failed: ${result.failed}`)
+```
+
+### forEach Options
+
+| Option | Type | Description |
+|--------|------|-------------|
+| `concurrency` | `number` | Max parallel operations (default: 1) |
+| `maxRetries` | `number` | Retries per item (default: 0) |
+| `retryDelay` | `number \| fn` | Delay between retries |
+| `onProgress` | `fn` | Progress callback |
+| `onError` | `'continue' \| 'retry' \| 'skip' \| 'stop' \| fn` | Error handling |
+| `timeout` | `number` | Timeout per item in ms |
+| `persist` | `boolean \| string` | Enable durability (string = custom action name) |
+| `resume` | `string` | Resume from action ID |
+
+### Durable forEach
+
+Persist progress to survive crashes:
+
+```typescript
+// Enable persistence - auto-names action as "Lead.forEach"
+const result = await db.Lead.forEach(processLead, {
+  concurrency: 10,
+  persist: true,
+})
+
+console.log(`Action ID: ${result.actionId}`)
+```
+
+Custom action name:
+
+```typescript
+await db.Lead.forEach(processLead, {
+  persist: 'analyze-leads',  // Custom action name
+})
+```
+
+Resume after a crash:
+
+```typescript
+await db.Lead.forEach(processLead, {
+  resume: 'action-123',  // Skips already-processed items
+})
+```
+
+---
+
+## Actions
+
+Track long-running operations:
+
+```typescript
+const action = await actions.create({
+  type: 'import-leads',
+  data: { file: 'leads.csv' },
+  total: 1000,
+})
+
+await actions.update(action.id, { progress: 500 })
+await actions.update(action.id, { status: 'completed' })
+```
+
+---
+
+## Installation
+
+```bash
+pnpm add ai-database
+```
+
+## Configuration
+
+```bash
+DATABASE_URL=./content         # filesystem (default)
+DATABASE_URL=sqlite://./data   # SQLite
+DATABASE_URL=:memory:          # in-memory
+```
+
+## Documentation
+
+- [Full Documentation](https://primitives.org.ai/database)
+- [CRUD Operations](https://primitives.org.ai/database/create)
+- [Schema Types](https://primitives.org.ai/database/schema)
+- [Events](https://primitives.org.ai/database/events)
+
+## Document Database Interface
+
+In addition to the schema-first graph model, `ai-database` also exports environment-agnostic types for document-based storage (MDX files with frontmatter). These types are used by `@mdxdb/*` adapters and work in any JavaScript runtime (Node.js, Bun, Deno, Workers, Browser).
+
+```typescript
+import type {
+  DocumentDatabase,
+  DocListOptions,
+  DocSearchOptions,
+  Document,
+} from 'ai-database'
+
+// The DocumentDatabase interface
+interface DocumentDatabase<TData> {
+  list(options?: DocListOptions): Promise<DocListResult<TData>>
+  search(options: DocSearchOptions): Promise<DocSearchResult<TData>>
+  get(id: string, options?: DocGetOptions): Promise<Document<TData> | null>
+  set(id: string, doc: Document<TData>, options?: DocSetOptions): Promise<DocSetResult>
+  delete(id: string, options?: DocDeleteOptions): Promise<DocDeleteResult>
+  close?(): Promise<void>
+}
+```
+
+### Document Types
+
+| Type | Description |
+|------|-------------|
+| `Document<TData>` | MDX document with id, type, context, data, and content |
+| `DocumentDatabase<TData>` | Interface for document storage adapters |
+| `DocListOptions` | Options for listing documents (limit, offset, sortBy, type, prefix) |
+| `DocListResult<TData>` | List result with documents, total, hasMore |
+| `DocSearchOptions` | Search options (query, fields, semantic) |
+| `DocSearchResult<TData>` | Search result with scores |
+| `DocGetOptions` | Get options (includeAst, includeCode) |
+| `DocSetOptions` | Set options (createOnly, updateOnly, version) |
+| `DocSetResult` | Set result (id, version, created) |
+| `DocDeleteOptions` | Delete options (soft, version) |
+| `DocDeleteResult` | Delete result (id, deleted) |
+
+### View Types
+
+For bi-directional relationship rendering:
+
+| Type | Description |
+|------|-------------|
+| `ViewManager` | Interface for managing views |
+| `ViewDocument` | View template definition |
+| `ViewContext` | Context for rendering a view |
+| `ViewRenderResult` | Rendered markdown and entities |
+| `ViewSyncResult` | Mutations from extracting edited markdown |
+| `DocumentDatabaseWithViews` | Database with view support |
+
+### Usage with @mdxdb adapters
+
+```typescript
+// Filesystem adapter
+import { createFsDatabase } from '@mdxdb/fs'
+const db = createFsDatabase({ root: './content' })
+
+// API adapter
+import { createApiDatabase } from '@mdxdb/api'
+const db = createApiDatabase({ baseUrl: 'https://api.example.com' })
+
+// SQLite adapter
+import { createSqliteDatabase } from '@mdxdb/sqlite'
+const db = createSqliteDatabase({ path: './data.db' })
+
+// Same DocumentDatabase interface regardless of backend
+const doc = await db.get('posts/hello-world')
+await db.set('posts/new', { data: { title: 'New Post' }, content: '# Hello' })
+```
 
-## API Reference
-[API documentation link]
+## Related
 
-## Dependencies
-Built on top of [mdxdb](https://github.com/ai-primitives/mdxdb) for robust document storage and vector search capabilities.
+- [ai-functions](https://github.com/ai-primitives/ai-primitives/tree/main/packages/ai-functions) - AI-powered functions
+- [ai-workflows](https://github.com/ai-primitives/ai-primitives/tree/main/packages/ai-workflows) - Event-driven workflows
+- [@mdxdb/fs](https://github.com/ai-primitives/mdx.org.ai/tree/main/packages/@mdxdb/fs) - Filesystem adapter
+- [@mdxdb/sqlite](https://github.com/ai-primitives/mdx.org.ai/tree/main/packages/@mdxdb/sqlite) - SQLite adapter
+- [@mdxdb/api](https://github.com/ai-primitives/mdx.org.ai/tree/main/packages/@mdxdb/api) - HTTP API client