latticesql 3.0.0 → 3.2.0

package/docs/examples/agent-system.md

@@ -0,0 +1,313 @@
+# Example: AI Agent System
+
+A complete example showing how to use `latticesql` as the persistent memory layer for a multi-agent AI system.
+
+---
+
+## Scenario
+
+You have several specialised AI agents (Craft, Audit, Research, etc.). Each agent has a persistent profile and an assigned task list. You want:
+
+- A SQLite database tracking agents, tasks, and events
+- Auto-generated LLM context files so each agent can read its current state
+- A writeback channel so agents can append new tasks by writing to a file
+- Audit logging for all database mutations
+
+---
+
+## Project structure
+
+```
+my-agent-system/
+├── lattice.config.yml
+├── src/
+│   └── db.ts
+├── context/
+│   ├── AGENTS.md
+│   └── TASKS.md
+├── data/
+│   └── agents.db
+└── generated/
+    ├── types.ts
+    └── migration.sql
+```
+
+---
+
+## 1. Define the schema
+
+```yaml
+# lattice.config.yml
+db: ./data/agents.db
+
+entities:
+  agent:
+    fields:
+      id: { type: uuid, primaryKey: true }
+      slug: { type: text, required: true }
+      name: { type: text, required: true }
+      role: { type: text, required: true }
+      status: { type: text, default: active }
+      model: { type: text, default: claude-sonnet-4-6 }
+      created_at: { type: datetime }
+    render: default-table
+    outputFile: context/AGENTS.md
+
+  task:
+    fields:
+      id: { type: uuid, primaryKey: true }
+      title: { type: text, required: true }
+      description: { type: text }
+      status: { type: text, default: pending }
+      priority: { type: integer, default: 1 }
+      agent_id: { type: uuid, ref: agent }
+      created_at: { type: datetime }
+      completed_at: { type: datetime }
+    render:
+      template: default-list
+      formatRow: '[{{status}}] {{title}} → {{agent.name}}'
+    outputFile: context/TASKS.md
+
+  event:
+    fields:
+      id: { type: uuid, primaryKey: true }
+      type: { type: text, required: true }
+      agent_id: { type: uuid, ref: agent }
+      payload: { type: text }
+      created_at: { type: datetime }
+    render:
+      template: default-list
+      formatRow: '{{created_at}} [{{type}}] {{agent.name}}'
+    outputFile: context/EVENTS.md
+```
+
+---
+
+## 2. Generate types and migration
+
+```sh
+npx lattice generate
+```
+
+Produces `generated/types.ts`:
+
+```ts
+export interface Agent {
+  id: string;
+  slug: string;
+  name: string;
+  role: string;
+  status?: string;
+  model?: string;
+  created_at?: string;
+}
+
+export interface Task {
+  id: string;
+  title: string;
+  description?: string;
+  status?: string;
+  priority?: number;
+  agent_id?: string; // → agent
+  created_at?: string;
+  completed_at?: string;
+}
+
+export interface Event {
+  id: string;
+  type: string;
+  agent_id?: string; // → agent
+  payload?: string;
+  created_at?: string;
+}
+```
+
+---
+
+## 3. Set up the database
+
+```ts
+// src/db.ts
+import { Lattice } from 'latticesql';
+
+export const db = new Lattice({ config: './lattice.config.yml' });
+
+db.on('audit', (event) => {
+  console.log(`[audit] ${event.operation} ${event.table}:${event.id} at ${event.timestamp}`);
+});
+
+db.on('render', (result) => {
+  console.log(`[render] Wrote ${result.filesWritten.length} file(s) in ${result.durationMs}ms`);
+});
+```
+
+---
+
+## 4. Seed agents
+
+```ts
+import { db } from './db.js';
+
+async function seedAgents() {
+  await db.init();
+
+  const agents = [
+    { id: 'agent-craft', slug: 'craft', name: 'Craft', role: 'Software architect and implementer' },
+    { id: 'agent-audit', slug: 'audit', name: 'Audit', role: 'Code reviewer and security analyst' },
+    {
+      id: 'agent-research',
+      slug: 'research',
+      name: 'Research',
+      role: 'Technical researcher and planner',
+    },
+  ];
+
+  for (const agent of agents) {
+    await db.upsert('agent', {
+      ...agent,
+      status: 'active',
+      model: 'claude-sonnet-4-6',
+      created_at: new Date().toISOString(),
+    });
+  }
+
+  console.log('Agents seeded.');
+}
+
+seedAgents();
+```
+
+---
+
+## 5. Assign and complete tasks
+
+```ts
+import { db } from './db.js';
+
+// Create a task
+const taskId = await db.insert('task', {
+  title: 'Add rate limiting to the API',
+  description: 'Implement token bucket rate limiting on /api/* routes',
+  status: 'pending',
+  priority: 2,
+  agent_id: 'agent-craft',
+  created_at: new Date().toISOString(),
+});
+
+// Query open tasks for an agent
+const craftTasks = await db.query('task', {
+  where: { agent_id: 'agent-craft', status: 'pending' },
+  orderBy: 'priority',
+  orderDir: 'desc',
+});
+
+// Mark a task complete
+await db.update('task', taskId, {
+  status: 'done',
+  completed_at: new Date().toISOString(),
+});
+
+// Count by status
+const openCount = await db.count('task', { where: { status: 'pending' } });
+const doneCount = await db.count('task', { where: { status: 'done' } });
+console.log(`Tasks: ${openCount} open, ${doneCount} done`);
+```
+
+---
+
+## 6. Writeback: let agents append tasks
+
+Agents can write to a shared inbox file. Lattice reads it back and inserts into the database.
+
+```ts
+import { db } from './db.js';
+
+// Register the writeback pipeline
+db.defineWriteback({
+  file: './context/INBOX.md',
+  parse: (content, fromOffset) => {
+    const newContent = content.slice(fromOffset);
+    const entries: { title: string; agentSlug: string }[] = [];
+
+    // Parse lines like: "- TASK: [agent-slug] Task title here"
+    for (const line of newContent.split('\n')) {
+      const match = line.match(/^- TASK: \[([^\]]+)\] (.+)$/);
+      if (match) {
+        entries.push({ agentSlug: match[1]!, title: match[2]! });
+      }
+    }
+
+    return { entries, nextOffset: content.length };
+  },
+  persist: async (entry) => {
+    const { title, agentSlug } = entry as { title: string; agentSlug: string };
+    // Look up the agent by slug
+    const [agent] = await db.query('agent', { where: { slug: agentSlug } });
+    await db.insert('task', {
+      title,
+      status: 'pending',
+      priority: 1,
+      agent_id: agent?.id ?? null,
+      created_at: new Date().toISOString(),
+    });
+  },
+  dedupeKey: (entry) => {
+    const { title, agentSlug } = entry as { title: string; agentSlug: string };
+    return `${agentSlug}:${title}`;
+  },
+});
+```
+
+Now any agent that writes `- TASK: [craft] Refactor auth middleware` to `context/INBOX.md` will have it automatically picked up on the next sync cycle.
+
+---
+
+## 7. Start the sync loop
+
+```ts
+import { db } from './db.js';
+
+await db.init();
+
+// Render once immediately:
+await db.render('./context');
+
+// Then watch with a 10-second interval:
+const stop = await db.watch('./context', {
+  interval: 10_000,
+  onError: (err) => console.error('Sync error:', err),
+});
+
+// Graceful shutdown:
+process.on('SIGTERM', () => {
+  stop();
+  db.close();
+});
+```
+
+---
+
+## 8. What the context files look like
+
+**`context/AGENTS.md`** (rendered by `default-table`):
+
+```markdown
+# agent
+
+| id          | slug  | name  | role               | status | model             |
+| ----------- | ----- | ----- | ------------------ | ------ | ----------------- |
+| agent-craft | craft | Craft | Software architect | active | claude-sonnet-4-6 |
+| agent-audit | audit | Audit | Code reviewer      | active | claude-sonnet-4-6 |
+```
+
+**`context/TASKS.md`** (rendered by `default-list` with `formatRow`):
+
+```markdown
+# task
+
+- [pending] Add rate limiting to the API → Craft
+- [done] Write migration guide → Audit
+- [pending] Research vector search options → Research
+```
+
+These files are what the agents read at the start of each conversation — structured, compact, and automatically kept in sync with the database.

package/docs/examples/cms.md

@@ -0,0 +1,366 @@
+# Example: Content Management System
+
+A complete example showing how to build a CMS with `latticesql` — authors, posts, tags, and generated context files for LLM-assisted content operations.
+
+---
+
+## Scenario
+
+A content team wants:
+
+- A SQLite-backed CMS storing authors, posts, and tags
+- Markdown context files that LLMs can read for drafting, editing, and summarisation tasks
+- A custom render function for the posts index (too complex for a built-in template)
+- A multi-table view that produces one context file per author
+
+---
+
+## Project structure
+
+```
+my-cms/
+├── lattice.config.yml
+├── src/
+│   ├── db.ts
+│   └── content.ts
+├── context/
+│   ├── AUTHORS.md
+│   ├── POSTS.md
+│   └── authors/
+│       ├── alice.md
+│       └── bob.md
+├── data/
+│   └── cms.db
+└── generated/
+    ├── types.ts
+    └── migration.sql
+```
+
+---
+
+## 1. Schema
+
+```yaml
+# lattice.config.yml
+db: ./data/cms.db
+
+entities:
+  author:
+    fields:
+      id: { type: uuid, primaryKey: true }
+      slug: { type: text, required: true }
+      name: { type: text, required: true }
+      email: { type: text, required: true }
+      bio: { type: text }
+      active: { type: boolean, default: 1 }
+    render: default-table
+    outputFile: context/AUTHORS.md
+
+  tag:
+    fields:
+      id: { type: uuid, primaryKey: true }
+      label: { type: text, required: true }
+    render: default-list
+    outputFile: context/TAGS.md
+
+  post:
+    fields:
+      id: { type: uuid, primaryKey: true }
+      slug: { type: text, required: true }
+      title: { type: text, required: true }
+      excerpt: { type: text }
+      status: { type: text, default: draft }
+      author_id: { type: uuid, ref: author }
+      word_count: { type: integer, default: 0 }
+      published_at: { type: datetime }
+      updated_at: { type: datetime }
+    render:
+      template: default-detail
+      formatRow: '**{{title}}** [{{status}}] by {{author.name}} — {{word_count}} words'
+    outputFile: context/POSTS.md
+```
+
+---
+
+## 2. Generate and migrate
+
+```sh
+npx lattice generate --out src/generated
+```
+
+Then apply the generated SQL to your database on first run.
+
+---
+
+## 3. Database and custom render
+
+The `post` entity uses the `default-detail` template with a `formatRow` hook from the YAML config. But for the per-author context files, we need a custom multi-table view that can't be expressed in YAML:
+
+```ts
+// src/db.ts
+import { Lattice } from 'latticesql';
+
+export const db = new Lattice({
+  config: './lattice.config.yml',
+  options: { wal: true },
+});
+
+// Per-author context files — one file per author
+db.defineMulti('author-context', {
+  keys: () => db.query('author', { where: { active: 1 } }),
+
+  outputFile: (author) => `context/authors/${author.slug as string}.md`,
+
+  tables: ['post'],
+
+  render: (author, tables) => {
+    const authorPosts = (tables.post ?? [])
+      .filter((p) => p.author_id === author.id)
+      .sort((a, b) => String(b.updated_at ?? '').localeCompare(String(a.updated_at ?? '')));
+
+    const published = authorPosts.filter((p) => p.status === 'published');
+    const drafts = authorPosts.filter((p) => p.status === 'draft');
+
+    const lines: string[] = [
+      `# ${author.name as string}`,
+      '',
+      author.bio ? `${author.bio as string}` : '',
+      '',
+      `**Published:** ${published.length} posts`,
+      `**Drafts:** ${drafts.length} posts`,
+      '',
+      '## Recent Posts',
+      '',
+      ...published
+        .slice(0, 5)
+        .map(
+          (p) =>
+            `- **${p.title as string}** — ${(p.published_at as string) ?? 'unpublished'} (${p.word_count as number} words)`,
+        ),
+    ];
+
+    if (drafts.length > 0) {
+      lines.push('', '## Drafts', '');
+      for (const draft of drafts) {
+        lines.push(`- ${draft.title as string} _(${draft.word_count as number} words)_`);
+      }
+    }
+
+    return lines.filter((l) => l !== null).join('\n');
+  },
+});
+
+await db.init({
+  migrations: [
+    {
+      version: 1,
+      sql: 'ALTER TABLE post ADD COLUMN featured INTEGER DEFAULT 0',
+    },
+  ],
+});
+```
+
+---
+
+## 4. Content operations
+
+```ts
+// src/content.ts
+import { db } from './db.js';
+
+// --- Authors ---
+
+export async function createAuthor(opts: {
+  slug: string;
+  name: string;
+  email: string;
+  bio?: string;
+}) {
+  return db.insert('author', {
+    ...opts,
+    bio: opts.bio ?? null,
+    active: 1,
+  });
+}
+
+// --- Posts ---
+
+export async function createDraft(opts: {
+  slug: string;
+  title: string;
+  excerpt?: string;
+  authorId: string;
+}) {
+  return db.insert('post', {
+    ...opts,
+    excerpt: opts.excerpt ?? null,
+    status: 'draft',
+    author_id: opts.authorId,
+    word_count: 0,
+    updated_at: new Date().toISOString(),
+  });
+}
+
+export async function updatePost(
+  id: string,
+  content: { title?: string; excerpt?: string; wordCount?: number },
+) {
+  await db.update('post', id, {
+    ...(content.title ? { title: content.title } : {}),
+    ...(content.excerpt ? { excerpt: content.excerpt } : {}),
+    ...(content.wordCount ? { word_count: content.wordCount } : {}),
+    updated_at: new Date().toISOString(),
+  });
+}
+
+export async function publishPost(id: string) {
+  await db.update('post', id, {
+    status: 'published',
+    published_at: new Date().toISOString(),
+    updated_at: new Date().toISOString(),
+  });
+}
+
+export async function getPublishedPosts(authorId?: string) {
+  if (authorId) {
+    return db.query('post', {
+      where: { status: 'published', author_id: authorId },
+      orderBy: 'published_at',
+      orderDir: 'desc',
+    });
+  }
+  return db.query('post', {
+    where: { status: 'published' },
+    orderBy: 'published_at',
+    orderDir: 'desc',
+  });
+}
+
+export async function searchPosts(query: string) {
+  return db.query('post', {
+    filters: [
+      { col: 'title', op: 'like', val: `%${query}%` },
+      { col: 'status', op: 'ne', val: 'archived' },
+    ],
+  });
+}
+
+export async function getLongPosts(minWords = 1000) {
+  return db.query('post', {
+    filters: [
+      { col: 'word_count', op: 'gte', val: minWords },
+      { col: 'status', op: 'eq', val: 'published' },
+    ],
+    orderBy: 'word_count',
+    orderDir: 'desc',
+  });
+}
+
+// --- Tags (upsert by label) ---
+
+export async function findOrCreateTag(label: string) {
+  return db.upsertBy('tag', 'label', label, {});
+}
+
+// --- Stats ---
+
+export async function getCMSStats() {
+  const [totalPosts, publishedPosts, draftPosts, totalAuthors] = await Promise.all([
+    db.count('post'),
+    db.count('post', { where: { status: 'published' } }),
+    db.count('post', { where: { status: 'draft' } }),
+    db.count('author', { where: { active: 1 } }),
+  ]);
+  return { totalPosts, publishedPosts, draftPosts, totalAuthors };
+}
+```
+
+---
+
+## 5. Sync loop
+
+```ts
+import { db } from './db.js';
+
+// Render on demand after writes:
+await db.render('./context');
+
+// Or watch with a longer interval (content changes slowly):
+const stop = await db.watch('./context', {
+  interval: 60_000, // 1 minute
+  onRender: (r) => {
+    if (r.filesWritten.length > 0) {
+      console.log(`[cms] Context updated: ${r.filesWritten.join(', ')}`);
+    }
+  },
+});
+```
+
+---
+
+## 6. Sample context output
+
+**`context/POSTS.md`** (rendered by `default-detail` with `formatRow`):
+
+```markdown
+# post
+
+## post-1
+
+**How Lattice Works** [published] by Alice — 1240 words
+
+## post-2
+
+**Getting Started with AI Agents** [published] by Bob — 980 words
+
+## post-3
+
+**Draft: Advanced Query Patterns** [draft] by Alice — 340 words
+```
+
+**`context/authors/alice.md`** (multi-table custom render):
+
+```markdown
+# Alice
+
+Senior technical writer with a focus on developer tooling.
+
+**Published:** 12 posts
+**Drafts:** 2 posts
+
+## Recent Posts
+
+- **How Lattice Works** — 2026-03-15 (1240 words)
+- **Building Context-Aware AI Systems** — 2026-03-01 (1800 words)
+
+## Drafts
+
+- Advanced Query Patterns _(340 words)_
+- Template Rendering Deep Dive _(120 words)_
+```
+
+An LLM given access to `context/authors/alice.md` immediately knows Alice's recent work, what she's drafted, and can assist with writing, editing, or summarising without querying the database.
+
+---
+
+## 7. Using the escape hatch for raw queries
+
+For complex queries not covered by the Lattice API — for example, joining posts to tags through a join table — use the `db.db` escape hatch:
+
+```ts
+// Get posts with their tag labels (join table not modelled in Lattice)
+const stmt = db.db.prepare(`
+  SELECT p.id, p.title, GROUP_CONCAT(t.label) as tags
+  FROM post p
+  LEFT JOIN post_tag pt ON pt.post_id = p.id
+  LEFT JOIN tag t ON t.id = pt.tag_id
+  WHERE p.status = 'published'
+  GROUP BY p.id
+  ORDER BY p.published_at DESC
+  LIMIT ?
+`);
+
+const recentWithTags = stmt.all(10) as Array<{ id: string; title: string; tags: string }>;
+```
+
+The escape hatch bypasses Lattice sanitization and audit logging — use it only for read-only analytics queries.