luca 1.1.2 → 3.0.0

package/docs/tutorials/11-contentbase.md

@@ -0,0 +1,191 @@
+---
+title: Contentbase - Markdown as a Database
+tags: [contentbase, contentdb, markdown, database, models, query, collections]
+---
+
+# Contentbase - Markdown as a Database
+
+Contentbase lets you treat folders of markdown files as queryable database collections. Define models with Zod schemas, extract structured data from frontmatter and content, and query it with a fluent API.
+
+## Setup
+
+```typescript
+import container from '@soederpop/luca'
+
+const db = container.feature('contentDb', { rootPath: './content' })
+const { defineModel, section, hasMany, belongsTo } = db.library
+```
+
+## Directory Structure
+
+```
+content/
+├── posts/
+│   ├── hello-world.md
+│   ├── getting-started.md
+│   └── advanced-tips.md
+├── authors/
+│   ├── alice.md
+│   └── bob.md
+└── tags/
+    ├── javascript.md
+    └── typescript.md
+```
+
+## Defining Models
+
+Models map to subdirectories and define the shape of your content:
+
+```typescript
+import { z } from 'zod'
+
+const Post = defineModel('Post', {
+  // Maps to content/posts/
+  prefix: 'posts',
+
+  // Frontmatter schema
+  meta: z.object({
+    title: z.string(),
+    date: z.string(),
+    status: z.enum(['draft', 'published', 'archived']),
+    author: z.string().optional(),
+    tags: z.array(z.string()).default([]),
+  }),
+
+  // Extract structured data from the markdown body
+  sections: {
+    summary: section('Summary', {
+      extract: (query) => query.select('paragraph')?.toString() || '',
+      schema: z.string(),
+    }),
+    codeExamples: section('Code Examples', {
+      extract: (query) => query.selectAll('code').map((n: any) => n.toString()),
+      schema: z.array(z.string()),
+    }),
+  },
+
+  // Relationships
+  relationships: {
+    author: belongsTo(() => Author, { key: 'meta.author' }),
+    tags: hasMany(() => Tag, { heading: 'Tags' }),
+  },
+})
+
+const Author = defineModel('Author', {
+  prefix: 'authors',
+  meta: z.object({
+    name: z.string(),
+    email: z.string().email(),
+    role: z.enum(['writer', 'editor', 'admin']),
+  }),
+  relationships: {
+    posts: hasMany(() => Post, { foreignKey: 'meta.author' }),
+  },
+})
+```
+
+## Registering and Loading
+
+```typescript
+db.register(Post)
+db.register(Author)
+await db.load()  // Parses all markdown files and builds the queryable index
+```
+
+## Querying
+
+Contentbase provides a fluent query API:
+
+```typescript
+// Fetch all posts
+const allPosts = await db.query(Post).fetchAll()
+
+// Filter by frontmatter fields
+const published = await db.query(Post)
+  .where('meta.status', 'published')
+  .fetchAll()
+
+// Multiple filters
+const recentPosts = await db.query(Post)
+  .where('meta.status', 'published')
+  .where('meta.tags', 'includes', 'javascript')
+  .fetchAll()
+
+// Get a single document by slug (filename without .md)
+const post = await db.query(Post).find('hello-world')
+```
+
+## Markdown File Format
+
+Each markdown file has YAML frontmatter and a body:
+
+```markdown
+---
+title: Hello World
+date: 2024-01-15
+status: published
+author: alice
+tags: [javascript, tutorial]
+---
+
+# Hello World
+
+This is the post content.
+
+## Summary
+
+A brief introduction to our blog.
+
+## Code Examples
+
+\`\`\`javascript
+console.log('Hello!')
+\`\`\`
+```
+
+## Use Cases
+
+- **Documentation sites** -- query and render docs with frontmatter metadata
+- **Blog engines** -- posts with authors, tags, categories
+- **Knowledge bases** -- structured content with relationships
+- **Project management** -- epics, stories, tasks as markdown with status tracking
+- **Configuration** -- human-readable config files that are also queryable
+
+## Full Example: Blog Engine
+
+```typescript
+import container from '@soederpop/luca'
+import { z } from 'zod'
+
+const db = container.feature('contentDb', { rootPath: './blog' })
+const { defineModel, section, hasMany } = db.library
+
+const Post = defineModel('Post', {
+  prefix: 'posts',
+  meta: z.object({
+    title: z.string(),
+    date: z.string(),
+    status: z.enum(['draft', 'published']),
+    tags: z.array(z.string()).default([]),
+  }),
+  sections: {
+    excerpt: section('Excerpt', {
+      extract: (q) => q.select('paragraph')?.toString() || '',
+      schema: z.string(),
+    }),
+  },
+})
+
+db.register(Post)
+await db.load()
+
+// Get published posts for the homepage
+const posts = await db.query(Post)
+  .where('meta.status', 'published')
+  .fetchAll()
+
+for (const post of posts) {
+  console.log(`${post.meta.title} (${post.meta.date})`)
+  console.log(`  ${post.sections.excerpt}`)
+}
+```

package/docs/tutorials/12-assistants.md

@@ -0,0 +1,215 @@
+---
+title: Building Assistants
+tags: [assistants, ai, openai, tools, hooks, conversation, CORE.md]
+---
+
+# Building Assistants
+
+Assistants are AI-powered conversational agents defined by a file-based convention. Each assistant lives in its own folder with a system prompt, tools, hooks, and documentation.
+
+## Directory Structure
+
+```
+assistants/my-assistant/
+├── CORE.md           # System prompt (required)
+├── tools.ts          # Tool definitions with Zod schemas
+├── hooks.ts          # Lifecycle event handlers
+└── docs/             # Internal documentation the assistant can search
+    ├── guide.md
+    └── faq.md
+```
+
+## CORE.md -- The System Prompt
+
+This is the assistant's personality and instructions. It's a markdown file that becomes the system message:
+
+```markdown
+# Customer Support Assistant
+
+You are a helpful customer support agent for Acme Corp. You help users with
+billing questions, account issues, and product information.
+
+Always research internal docs before answering product questions.
+Be polite and concise.
+```
+
+## tools.ts -- Tool Definitions
+
+Define functions that the assistant can call. Each tool has a Zod schema describing its parameters:
+
+```typescript
+const { z } = require('zod')
+
+async function lookupOrder({ orderId }) {
+  // In a real app, query your database
+  return {
+    orderId,
+    status: 'shipped',
+    trackingNumber: 'ABC123',
+    estimatedDelivery: '2024-01-20',
+  }
+}
+
+async function createTicket({ subject, priority, description }) {
+  // Create a support ticket
+  const ticketId = `TICKET-${Date.now()}`
+  return {
+    ticketId,
+    subject,
+    priority,
+    message: `Ticket ${ticketId} created successfully`,
+  }
+}
+
+async function searchProducts({ query, category }) {
+  // Search product catalog
+  return {
+    results: [
+      { name: 'Widget Pro', price: 29.99, inStock: true },
+      { name: 'Widget Lite', price: 19.99, inStock: false },
+    ],
+  }
+}
+
+const schemas = {
+  lookupOrder: z.object({
+    orderId: z.string().describe('The order ID to look up'),
+  }).describe('Look up an order by its ID to get status and tracking info'),
+
+  createTicket: z.object({
+    subject: z.string().describe('Brief ticket subject line'),
+    priority: z.enum(['low', 'medium', 'high']).describe('Ticket priority'),
+    description: z.string().describe('Detailed description of the issue'),
+  }).describe('Create a new support ticket'),
+
+  searchProducts: z.object({
+    query: z.string().describe('Search terms'),
+    category: z.string().optional().describe('Product category filter'),
+  }).describe('Search the product catalog'),
+}
+
+module.exports = { lookupOrder, createTicket, searchProducts, schemas }
+```
+
+**Important:** The function name must match the key in the `schemas` object. The `.describe()` on the schema object itself becomes the tool description that the AI model sees.
+
+## hooks.ts -- Lifecycle Hooks
+
+React to assistant events:
+
+```typescript
+function started() {
+  console.log('[assistant] Session started')
+}
+
+function response(text) {
+  // Called when the assistant produces a text response
+  console.log(`[assistant] ${text.slice(0, 100)}...`)
+}
+
+function toolCall(name, args) {
+  // Called before a tool is executed
+  console.log(`[assistant] Calling tool: ${name}`, args)
+}
+
+function error(err) {
+  console.error('[assistant] Error:', err.message)
+}
+
+module.exports = { started, response, toolCall, error }
+```
+
+## docs/ -- Internal Documentation
+
+The `docs/` folder contains markdown files that the assistant can search using the built-in `researchInternalDocs` tool. This is automatically injected -- you don't need to define it in tools.ts.
+
+```
+docs/
+├── billing-faq.md
+├── product-catalog.md
+├── return-policy.md
+└── troubleshooting.md
+```
+
+In CORE.md, instruct the assistant to use it:
+
+```markdown
+When asked about products, billing, or policies, always use the
+researchInternalDocs tool first to find accurate information before answering.
+```
+
+## Using the Assistant
+
+### In a Script
+
+```typescript
+import container from '@soederpop/luca'
+
+const assistant = container.feature('assistant', {
+  folder: 'assistants/my-assistant',
+  model: 'gpt-4o',
+})
+
+// Ask a question
+const answer = await assistant.ask('What is the return policy?')
+console.log(answer)
+
+// Multi-turn conversation
+const follow = await assistant.ask('And how long does the refund take?')
+console.log(follow)
+
+// Save the conversation
+await assistant.save({ title: 'Return policy inquiry' })
+```
+
+### In an Endpoint
+
+Expose the assistant as an API:
+
+```typescript
+// endpoints/ask.ts
+import { z } from 'zod'
+import type { EndpointContext } from '@soederpop/luca'
+
+export const path = '/api/ask'
+export const description = 'Ask the support assistant a question'
+export const tags = ['assistant']
+
+export const postSchema = z.object({
+  question: z.string().describe('Your question'),
+})
+
+export async function post(params: z.infer<typeof postSchema>, ctx: EndpointContext) {
+  const assistant = ctx.container.feature('assistant', {
+    folder: 'assistants/my-assistant',
+    model: 'gpt-4o',
+  })
+
+  const answer = await assistant.ask(params.question)
+  return { answer }
+}
+```
+
+### Streaming Responses
+
+```typescript
+const assistant = container.feature('assistant', {
+  folder: 'assistants/my-assistant',
+  model: 'gpt-4o',
+})
+
+// Listen for chunks as they arrive
+assistant.on('chunk', (text) => {
+  process.stdout.write(text)
+})
+
+await assistant.ask('Explain quantum computing')
+```
+
+## Best Practices
+
+1. **Write focused CORE.md prompts** -- tell the assistant exactly what it is and what it should/shouldn't do
+2. **Keep tools simple** -- each tool should do one thing. The AI model is better at composing simple tools than using complex ones
+3. **Use docs/ liberally** -- put all reference material in docs/ so the assistant can look things up rather than relying on the model's training data
+4. **Use Zod `.describe()`** -- the descriptions on schemas and fields are what the model sees to decide when and how to call tools
+5. **Test with real questions** -- ask the assistant the kinds of things real users will ask

package/docs/tutorials/13-introspection.md

@@ -0,0 +1,157 @@
+---
+title: Introspection and Discovery
+tags: [introspection, runtime, discovery, documentation, describe, inspect]
+---
+
+# Introspection and Discovery
+
+One of Luca's defining features is that everything is discoverable at runtime. You don't need to read documentation to learn what's available -- you can ask the system itself.
+
+## Why Introspection Matters
+
+Introspection serves two audiences:
+
+1. **Developers** -- discover APIs while coding, without leaving the REPL or editor
+2. **AI Agents** -- learn the full API surface dynamically, enabling them to use features they weren't explicitly trained on
+
+## Container-Level Introspection
+
+```typescript
+// Structured data about the entire container
+const info = container.introspect()
+// Returns: registries, enabled features, state schema, available helpers
+
+// Human-readable markdown
+const docs = container.introspectAsText()
+```
+
+## Registry-Level Discovery
+
+```typescript
+// What's available?
+container.features.available
+// => ['fs', 'git', 'proc', 'vm', 'ui', 'diskCache', 'contentDb', ...]
+
+// Describe one
+container.features.describe('diskCache')
+// => Markdown documentation for diskCache feature
+
+// Describe everything
+container.features.describeAll()
+// => Full documentation for all registered features
+
+// Structured introspection data
+container.features.introspect('fs')
+// => { methods, getters, state, options, events, ... }
+```
+
+Same API for all registries:
+
+```typescript
+container.servers.available
+container.servers.describe('express')
+
+container.clients.available
+container.clients.describe('rest')
+
+container.commands.available
+container.commands.describe('serve')
+```
+
+## Helper-Level Introspection
+
+Every helper instance can describe itself:
+
+```typescript
+const fs = container.feature('fs')
+
+// Structured data
+const info = fs.introspect()
+// => { className, methods: [...], getters: [...], state: {...}, events: [...] }
+
+// Human-readable markdown
+const docs = fs.introspectAsText()
+```
+
+### Quick Discovery with $getters and $methods
+
+Every helper exposes `$getters` and `$methods` — string arrays listing what's available on the instance. Useful for quick exploration without parsing the full introspection object:
+
+```typescript
+const fs = container.feature('fs')
+fs.$methods  // => ['readFile', 'writeFile', 'walk', 'readdir', ...]
+fs.$getters  // => ['cwd', 'sep', ...]
+```
+
+### What's in the Introspection Data?
+
+- **Class name** and description (from JSDoc)
+- **Methods** -- name, description, parameters, return type
+- **Getters** -- name, description, type
+- **State schema** -- all observable state fields with descriptions
+- **Options schema** -- all configuration options with descriptions and defaults
+- **Events** -- known event names with descriptions
+
+## How It Works
+
+Introspection comes from two sources:
+
+1. **Build-time extraction** -- Luca's build step parses JSDoc comments, method signatures, and getter types from source code using AST analysis. Run `bun run build:introspection` to update this.
+
+2. **Runtime Zod schemas** -- State, options, and events schemas provide descriptions, types, and defaults at runtime via Zod's `.describe()` method.
+
+## Practical Example: Dynamic Tool Generation
+
+An AI agent can use introspection to generate tool definitions for any feature:
+
+```typescript
+// Agent discovers available features
+const available = container.features.available
+
+// Agent learns about a specific feature
+const fsInfo = container.features.introspect('fs')
+
+// fsInfo.methods tells the agent:
+// - readFile(path: string): string
+// - writeFile(path: string, content: string): Promise<string>
+// - walk(basePath: string, options?: WalkOptions): { files: string[], directories: string[] }
+// etc.
+
+// The agent can now use these methods without prior training on the fs feature
+```
+
+## Using Introspection in Your Features
+
+Make your custom features introspectable by:
+
+1. Writing JSDoc on the class, methods, and getters
+2. Using Zod `.describe()` on schema fields
+3. Running `bun run build:introspection` after changes
+
+```typescript
+/**
+ * Manages a pool of database connections with automatic health checking.
+ * Connections are recycled when they become stale or unhealthy.
+ */
+export class ConnectionPool extends Feature<PoolState, PoolOptions> {
+  /**
+   * Acquire a connection from the pool.
+   * Blocks until a connection is available or the timeout is reached.
+   */
+  async acquire(timeout?: number): Promise<Connection> {
+    // ...
+  }
+
+  /** The number of idle connections currently in the pool */
+  get idleCount(): number {
+    // ...
+  }
+
+  /** The number of active connections currently checked out */
+  get activeCount(): number {
+    // ...
+  }
+}
+```
+
+Now `container.features.describe('connectionPool')` returns rich documentation, and `container.features.introspect('connectionPool')` returns structured data -- all extracted from what you already wrote.