luca 1.1.2 → 3.0.0

package/docs/examples/yaml.md

@@ -0,0 +1,104 @@
+---
+title: "YAML"
+tags: [yaml, parsing, serialization, config]
+lastTested: null
+lastTestPassed: null
+---
+
+# yaml
+
+Parse YAML strings into JavaScript objects and serialize objects back to YAML. A thin wrapper around js-yaml.
+
+## Overview
+
+The `yaml` feature is on-demand. It provides two methods: `parse()` and `stringify()`. Use it any time you need to read or write YAML configuration files, convert between formats, or work with YAML data in memory.
+
+## Parsing a YAML String
+
+Start by enabling the feature and parsing some YAML.
+
+```ts
+const yml = container.feature('yaml')
+const config = yml.parse(`
+name: my-app
+version: 2.1.0
+database:
+  host: localhost
+  port: 5432
+features:
+  - auth
+  - logging
+  - caching
+`)
+console.log('Parsed name:', config.name)
+console.log('Parsed db host:', config.database.host)
+console.log('Parsed features:', config.features)
+```
+
+The parser handles nested objects, arrays, numbers, and booleans automatically.
+
+## Serializing an Object to YAML
+
+Use `stringify()` to convert a JavaScript object into a YAML-formatted string.
+
+```ts
+const output = yml.stringify({
+  server: { host: '0.0.0.0', port: 3000 },
+  logging: { level: 'info', pretty: true },
+  cors: { origins: ['https://example.com', 'https://app.example.com'] }
+})
+console.log('YAML output:')
+console.log(output)
+```
+
+The output is human-readable and suitable for writing to configuration files.
+
+## Round-Trip Conversion
+
+A common pattern is reading YAML, modifying data, and writing it back. Here we verify that a round-trip preserves data.
+
+```ts
+const original = `
+environment: production
+replicas: 3
+resources:
+  cpu: 500m
+  memory: 256Mi
+`
+const parsed = yml.parse(original)
+parsed.replicas = 5
+parsed.resources.memory = '512Mi'
+const updated = yml.stringify(parsed)
+console.log('Updated YAML:')
+console.log(updated)
+const reparsed = yml.parse(updated)
+console.log('Replicas after round-trip:', reparsed.replicas)
+console.log('Memory after round-trip:', reparsed.resources.memory)
+```
+
+The data survives the parse-modify-stringify cycle intact.
+
+## Working with Complex Structures
+
+YAML handles deeply nested and mixed-type structures well.
+
+```ts
+const complex = yml.stringify({
+  users: [
+    { name: 'Alice', roles: ['admin', 'editor'], active: true },
+    { name: 'Bob', roles: ['viewer'], active: false },
+  ],
+  settings: {
+    maxRetries: 3,
+    timeout: null,
+    nested: { deep: { value: 42 } }
+  }
+})
+console.log(complex)
+```
+
+Nulls, booleans, numbers, and nested arrays all serialize cleanly.
+
+## Summary
+
+This demo covered parsing YAML strings, serializing objects to YAML, round-trip conversion, and handling complex nested structures. The `yaml` feature gives you a clean two-method API for all YAML operations.

package/docs/ideas/assistant-factory-pattern.md

@@ -0,0 +1,142 @@
+---
+title: Assistant Factory Pattern — Eliminate Wrapper Duplication
+status: idea
+tags: [agi, assistant, refactor, lucaCoder, autonomousAssistant]
+---
+
+# Assistant Factory Pattern
+
+## Problem
+
+`LucaCoder` and `AutonomousAssistant` both duplicate the same pattern: create an inner `Assistant`, stack tool bundles, wire a permission interceptor, forward events. The permission system, approval flow, and event forwarding are copy-pasted between them. `LucaCoder` is just `AutonomousAssistant` + coding opinions (bash tool, skill loading, project instructions).
+
+Both features **wrap** an assistant rather than **composing** one. This means every method on `Assistant` (ask, tools, messages, conversation) has to be proxied through the wrapper.
+
+## Proposal
+
+### 1. Extract the permission layer into a `use()`-able plugin
+
+The permission system (allow/ask/deny per tool, pending approvals, approval history) is just a `beforeToolCall` interceptor. It doesn't need to be a feature — it's a plugin function.
+
+```typescript
+import type { Assistant } from './assistant'
+
+interface PermissionConfig {
+  permissions?: Record<string, 'allow' | 'ask' | 'deny'>
+  defaultPermission?: 'allow' | 'ask' | 'deny'
+}
+
+function withPermissions(config: PermissionConfig = {}) {
+  return (assistant: Assistant) => {
+    const perms = config.permissions || {}
+    const defaultPerm = config.defaultPermission || 'ask'
+    const pendingResolvers = new Map<string, (d: 'approve' | 'deny') => void>()
+
+    assistant.intercept('beforeToolCall', async (ctx, next) => {
+      const policy = perms[ctx.name] || defaultPerm
+
+      if (policy === 'deny') {
+        ctx.skip = true
+        ctx.result = JSON.stringify({ blocked: true, tool: ctx.name, reason: 'Permission denied.' })
+        assistant.emit('toolBlocked', ctx.name, 'deny policy')
+        return
+      }
+
+      if (policy === 'allow') {
+        await next()
+        return
+      }
+
+      // 'ask' — emit event and block until resolved
+      const id = assistant.container.utils.uuid()
+      const decision = await new Promise<'approve' | 'deny'>((resolve) => {
+        pendingResolvers.set(id, resolve)
+        assistant.emit('permissionRequest', { id, toolName: ctx.name, args: ctx.args })
+      })
+
+      pendingResolvers.delete(id)
+
+      if (decision === 'approve') {
+        await next()
+      } else {
+        ctx.skip = true
+        ctx.result = JSON.stringify({ blocked: true, tool: ctx.name, reason: 'User denied.' })
+      }
+    })
+
+    // Expose approve/deny on the assistant instance
+    ;(assistant as any).approve = (id: string) => pendingResolvers.get(id)?.('approve')
+    ;(assistant as any).deny = (id: string) => pendingResolvers.get(id)?.('deny')
+  }
+}
+```
+
+### 2. LucaCoder becomes an assistant factory, not a wrapper
+
+Instead of owning an inner assistant and proxying everything, `LucaCoder` creates and returns a configured `Assistant`:
+
+```typescript
+export class LucaCoder extends Feature {
+  createAssistant(overrides?: Partial<AssistantOptions>): Assistant {
+    const assistant = this.container.feature('assistant', {
+      systemPrompt: this.buildSystemPrompt(),
+      ...overrides,
+    })
+
+    // This is the runtime equivalent of hooks.ts started()
+    assistant.use(this.container.feature('codingTools'))
+
+    const fileTools = this.container.feature('fileTools')
+    assistant.use(fileTools.toTools({ only: ['editFile', 'writeFile', 'deleteFile'] }))
+    fileTools.setupToolsConsumer(assistant)
+
+    assistant.use(this.container.feature('processManager'))
+    assistant.use(this.container.feature('skillsLibrary'))
+
+    // Permission layer as a plugin
+    if (this.options.permissions || this.options.defaultPermission) {
+      assistant.use(withPermissions({
+        permissions: this.options.permissions,
+        defaultPermission: this.options.defaultPermission,
+      }))
+    }
+
+    return assistant
+  }
+}
+```
+
+### 3. Usage
+
+```typescript
+// Runtime — no disk files, works in compiled binary
+const coder = container.feature('lucaCoder')
+const assistant = await coder.createAssistant()
+await assistant.ask('refactor the auth module')
+
+// With permission gating
+const coder = container.feature('lucaCoder', {
+  permissions: { editFile: 'ask', writeFile: 'ask', deleteFile: 'ask' },
+  defaultPermission: 'allow',
+})
+const assistant = await coder.createAssistant()
+assistant.on('permissionRequest', ({ id }) => assistant.approve(id))
+```
+
+### 4. AutonomousAssistant collapses
+
+`AutonomousAssistant` becomes either:
+- Deleted entirely (replaced by `withPermissions` plugin on any assistant)
+- Or a thin factory like LucaCoder but with no coding opinions — just `createAssistant()` + permissions
+
+## What This Achieves
+
+- **No duplication** — permission logic lives in one place (the plugin)
+- **No proxying** — you get back a real `Assistant`, not a wrapper that forwards `.ask()`, `.tools`, `.messages`, `.conversation`
+- **Binary-compatible** — the factory pattern doesn't need disk files, so it works in the compiled `luca` binary
+- **Composable** — `assistants/codingAssistant/hooks.ts` and `LucaCoder.createAssistant()` produce the exact same result through the same mechanism (`assistant.use()`)
+- **The folder-based assistant becomes the reference implementation** of what the factory does programmatically
+
+## Relationship to assistants/codingAssistant
+
+The disk-based `assistants/codingAssistant/` (CORE.md + hooks.ts + tools.ts) is the "editable" version. `LucaCoder.createAssistant()` is the "compiled" version. Same tools, same prompt, same behavior — one loads from files, the other is baked in code.

package/docs/in-memory-fs.md

@@ -0,0 +1,4 @@
+# In Memory FS
+
+If we rigorously ensured all file system operations of other features, servers, clients, etc 
+went through the container.fs system, we could very easily build in memory file systems

package/docs/introspection-audit.md

@@ -0,0 +1,49 @@
+# Introspection Audit
+
+This audit is a periodic test, does the introspectAsText output actually help an LLM use that
+particular feature.
+
+## Files
+
+- `src/node/features/*.ts` NodeContainer features
+- `src/agi/features/*.ts` AGIContainer features
+
+## Your Task
+
+For each feature, you can run the following shell command:
+
+```shell
+luca describe feature-id
+```
+
+This command will accept diskCache, disk-cache.  Generally you want to go with the shortcut that is defined on the feature class.
+
+Your job is to take what you learn from that command, and build an idea of 3-4 example usecases of that feature.
+
+Compile all of this into a document in docs/reports/introspection-audit-tasks.md.
+
+I'd like the following format
+
+```
+# Introspection Audit Results
+
+## Features
+
+### [diskCache](src/node/features/disk-cache.ts)
+
+The purpose of this feature is to provided a consistent disk backed key-value store for large blobs and json objects.  It includes the ability to have an encrypted store that can only be read with a secret.
+
+- securely set / get a key across processes
+- query information about available keys
+- check if a key is present, if not write to it
+```
+
+Do that for every feature.
+
+## Next Steps
+
+Once we have this report written, I will review the use cases and descriptions to see how accurate they are and if I like them.
+
+Then we will have you attempt to eval your code and see if it works. If it doesn't, sugggest how the documentation could be improved or how the API might be improved.
+
+

package/docs/introspection.md

@@ -0,0 +1,164 @@
+# Introspection
+
+Luca's introspection system lets you discover everything about a container and its helpers at runtime. This document is a runnable demo — every code block works in the Luca REPL or as a script.
+
+## Container Introspection
+
+The container knows what it is, what registries it has, and what's available in each one.
+
+```ts
+const info = container.introspect()
+console.log(info.className)
+console.log(info.registries.map(r => `${r.name}: ${r.available.length} available`))
+console.log('factories:', info.factoryNames)
+console.log('enabled features:', info.enabledFeatures)
+```
+
+You can get the full introspection as markdown:
+
+```ts
+console.log(container.introspectAsText())
+```
+
+Or request a single section:
+
+```ts
+console.log(container.introspectAsText('methods'))
+```
+
+```ts
+console.log(container.introspectAsText('getters'))
+```
+
+## Querying Registries
+
+Every registry exposes what's available and can describe its members.
+
+```ts
+console.log('features:', container.features.available)
+```
+
+```ts
+console.log('clients:', container.clients.available)
+```
+
+```ts
+console.log('servers:', container.servers.available)
+```
+
+Describe a single member — returns markdown documentation derived from the code:
+
+```ts
+console.log(container.features.describe('git'))
+```
+
+Describe everything in a registry at once:
+
+```ts
+const allFeatureDocs = container.features.describeAll()
+console.log(`${allFeatureDocs.length} features documented`)
+console.log(allFeatureDocs[0].slice(0, 200) + '...')
+```
+
+## Helper Introspection — Quick Discovery
+
+Every helper instance exposes `$methods` and `$getters` for a quick look at what's available:
+
+```ts
+const git = container.feature('git')
+console.log('methods:', git.$methods)
+console.log('getters:', git.$getters)
+```
+
+## Helper Introspection — Structured Data
+
+Every helper instance (feature, client, server, etc.) can introspect itself. The result is a typed object you can traverse programmatically.
+
+```ts
+const git = container.feature('git')
+const data = git.introspect()
+console.log(data.id)
+console.log(data.description.slice(0, 100) + '...')
+console.log('methods:', Object.keys(data.methods))
+console.log('getters:', Object.keys(data.getters))
+```
+
+### Filtering by Section
+
+Pass a section name to get just that part:
+
+```ts
+const git = container.feature('git')
+const methodsOnly = git.introspect('methods')
+console.log(Object.keys(methodsOnly.methods))
+console.log(Object.keys(methodsOnly.getters)) // empty — filtered out
+```
+
+Valid sections: `'methods'`, `'getters'`, `'events'`, `'state'`, `'options'`.
+
+### Expanded Type Properties
+
+When a method parameter uses a custom type, introspection resolves it to show the type's members:
+
+```ts
+const git = container.feature('git')
+const lsFiles = git.introspect('methods').methods.lsFiles
+console.log(lsFiles.parameters.options.type) // "LsFilesOptions"
+console.log(lsFiles.parameters.options.properties)
+// { cached: { type: 'boolean', description: 'Show cached/staged files' }, ... }
+```
+
+## Helper Introspection — As Text
+
+Get the full markdown documentation for a helper:
+
+```ts
+const git = container.feature('git')
+console.log(git.introspectAsText())
+```
+
+Or just one section:
+
+```ts
+const git = container.feature('git')
+console.log(git.introspectAsText('methods'))
+```
+
+```ts
+const git = container.feature('git')
+console.log(git.introspectAsText('getters'))
+```
+
+The heading depth can be controlled — useful when embedding in larger documents:
+
+```ts
+const git = container.feature('git')
+console.log(git.introspectAsText('methods', 3)) // headings start at ###
+```
+
+## Introspecting State and Options
+
+Features that declare Zod schemas for state and options expose them through introspection:
+
+```ts
+const git = container.feature('git')
+const stateInfo = git.introspect('state')
+console.log(stateInfo.state)
+```
+
+```ts
+const ui = container.feature('ui')
+const stateInfo = ui.introspect('state')
+console.log(stateInfo.state)
+```
+
+## Putting It Together
+
+A quick way to survey everything the container offers:
+
+```ts
+for (const name of container.registryNames) {
+  const registry = container[name]
+  console.log(`\n${name}: ${registry.available.join(', ')}`)
+}
+```

package/docs/mcp/readme.md

@@ -0,0 +1,162 @@
+# Luca Development Guide
+
+You are working in a **luca project**. The luca container provides all capabilities your code needs. Do not install npm packages or import Node.js builtins directly.
+
+## The Contract
+
+Every capability goes through the container. If you need something that doesn't exist, build it as a feature, client, or server. If it wraps a third-party library, the helper IS the interface — consumer code never imports the library directly.
+
+## Import Rule
+
+All consumer code imports from `@soederpop/luca` only:
+
+```ts
+import { Feature, features, z, FeatureStateSchema, FeatureOptionsSchema } from '@soederpop/luca'
+import { Client, clients, RestClient, ClientStateSchema } from '@soederpop/luca/client'
+import { Server, servers, ServerStateSchema } from '@soederpop/luca'
+import { commands, CommandOptionsSchema } from '@soederpop/luca'
+```
+
+Never import from `fs`, `path`, `crypto`, or other Node builtins. Never import third-party packages in consumer code. If a container feature wraps the functionality, use it.
+
+## Zod v4
+
+This project uses **Zod v4** — import `z` from `@soederpop/luca`, never from `'zod'` directly. All option, state, and event schemas use Zod v4 syntax. Key patterns:
+
+```ts
+// Extending base schemas (options, state, events)
+export const MyStateSchema = FeatureStateSchema.extend({
+  count: z.number().default(0).describe('Number of items'),
+  label: z.string().optional().describe('Display label'),
+})
+
+// Events use z.tuple() for listener arguments
+export const MyEventsSchema = FeatureEventsSchema.extend({
+  itemAdded: z.tuple([z.string().describe('key'), z.number().describe('index')]),
+})
+
+// Type inference
+export type MyState = z.infer<typeof MyStateSchema>
+```
+
+Zod v4 differences from v3 that matter:
+- `z.string().check(...)` replaces some v3 refinement patterns
+- `.toJSONSchema()` is built-in on any schema — no external library needed
+- Error customization uses `z.string({ error: "message" })` not `.refine()` for simple cases
+- `z.interface()` exists for recursive/lazy object types
+- Do NOT use `z.nativeEnum()` — use `z.enum()` instead
+
+## Dependencies
+
+If the project has `node_modules` and a package manager, helper implementations can import third-party libraries internally. If not (e.g. running via the `luca` binary's VM), all code must import only from `@soederpop/luca`.
+
+## Discovering Capabilities
+
+The container has registries for features, clients, servers, commands, and endpoints. **Do not guess** what is available — use your MCP tools to discover it:
+
+1. **`find_capability`** — Overview of all features, clients, and servers with descriptions. Start here.
+2. **`list_registry`** — List all names in a specific registry (features, clients, servers, commands, endpoints).
+3. **`describe_helper`** — Full API docs for a specific helper (methods, options, state, events). Call this before writing code that uses a helper.
+4. **`eval`** — Once you know what you need, prototype calls in the live sandbox before writing them into files.
+
+## Mini Examples
+
+### Feature with composition
+
+Features access other features via `this.container.feature(...)`:
+
+```ts
+import { z } from 'zod'
+import { FeatureStateSchema, FeatureOptionsSchema, FeatureEventsSchema } from '@soederpop/luca'
+import { Feature, features } from '@soederpop/luca'
+import type { ContainerContext } from '@soederpop/luca'
+
+export const ConfigStateSchema = FeatureStateSchema.extend({
+  loaded: z.boolean().default(false).describe('Whether config has been loaded'),
+})
+
+export const ConfigOptionsSchema = FeatureOptionsSchema.extend({
+  filePath: z.string().default('config.json').describe('Path to config file'),
+})
+
+export const ConfigEventsSchema = FeatureEventsSchema.extend({
+  configLoaded: z.tuple([z.record(z.unknown()).describe('parsed config')]),
+})
+
+export class Config extends Feature<z.infer<typeof ConfigStateSchema>, z.infer<typeof ConfigOptionsSchema>> {
+  static override shortcut = 'features.config' as const
+  static override stateSchema = ConfigStateSchema
+  static override optionsSchema = ConfigOptionsSchema
+  static override eventsSchema = ConfigEventsSchema
+  static override description = 'Loads and manages project configuration'
+
+  /** Load and parse the config file */
+  async load() {
+    const fs = this.container.feature('fs')
+    const raw = await fs.readFile(this.options.filePath)
+    const data = JSON.parse(raw)
+    this.state.set('loaded', true)
+    this.emit('configLoaded', data)
+    return data
+  }
+}
+
+declare module '@soederpop/luca' {
+  interface AvailableFeatures { config: typeof Config }
+}
+export default features.register('config', Config)
+```
+
+### Client with composition
+
+Clients access features and other clients via `this.container`:
+
+```ts
+import { z } from 'zod'
+import { Client, clients, ClientStateSchema, ClientOptionsSchema, ClientEventsSchema } from '@soederpop/luca'
+import type { ContainerContext } from '@soederpop/luca'
+
+export const GithubOptionsSchema = ClientOptionsSchema.extend({
+  token: z.string().describe('GitHub personal access token'),
+  owner: z.string().describe('Repository owner'),
+  repo: z.string().describe('Repository name'),
+})
+
+export const GithubEventsSchema = ClientEventsSchema.extend({
+  issuesFetched: z.tuple([z.number().describe('count')]),
+})
+
+export class GithubClient extends Client<z.infer<typeof ClientStateSchema>, z.infer<typeof GithubOptionsSchema>> {
+  static override shortcut = 'clients.github' as const
+  static override optionsSchema = GithubOptionsSchema
+  static override eventsSchema = GithubEventsSchema
+  static override description = 'GitHub API client using container REST client'
+
+  /** Fetch open issues */
+  async issues() {
+    const rest = this.container.client('rest')
+    const res = await rest.get(`https://api.github.com/repos/${this.options.owner}/${this.options.repo}/issues`, {
+      headers: { Authorization: `token ${this.options.token}` },
+    })
+    this.emit('issuesFetched', res.data.length)
+    return res.data
+  }
+}
+
+declare module '@soederpop/luca' {
+  interface AvailableClients { github: typeof GithubClient }
+}
+export default clients.register('github', GithubClient)
+```
+
+## Workflow
+
+1. **`find_capability`** — Search what already exists before writing anything
+2. **`describe_helper`** — Read the full API docs for the helper you need
+3. **`eval`** — Prototype and test container API calls in the sandbox
+4. **`scaffold`** — Generate correct boilerplate when building something new
+5. **Write the file** — Using the patterns from the scaffold
+
+## Portability
+
+Code that only imports from `@soederpop/luca` can be copied between any luca project. That's the goal. Features, clients, servers, and commands written this way are portable building blocks.

package/docs/models.ts

@@ -0,0 +1,41 @@
+import {
+  defineModel,
+  section,
+  hasMany,
+  belongsTo,
+  AstQuery,
+  z,
+} from "contentbase";
+
+export const Tutorial = defineModel("Tutorial", {
+	description: 'Tutorials on how to compose things together to do neat things',
+	prefix: "tutorials",
+	meta: z.object({
+		tags: z.array(z.string()).default([]).describe("Arbitrary tags for categorizing the tutorial"),
+	}),
+})
+
+export const Report = defineModel("Report", {
+	description: 'Used for e.g. documentation audits, usability audits for agents, or anything else long form project related',
+	prefix: "reports",
+	meta: z.object({
+		tags: z.array(z.string()).default([]).describe("Arbitrary tags for categorizing the report"),
+	}),
+})
+
+export const Example = defineModel("Example", {
+	description: 'Examples of using various luca features',
+	prefix: "examples",
+	meta: z.object({
+		tags: z.array(z.string()).default([]).describe("Arbitrary tags for categorizing the example"),
+	}),
+})
+
+export const Challenge = defineModel('Challenge', {
+  description: 'challenges are used by our evaluation suite to measure the quality of the introspection content and tool, as well as the SKILL.md that gets generated to help coding assistants work with the luca framework',
+  prefix: 'challenges',
+  meta: z.object({
+	 difficulty: z.enum(['easy','medium','hard']).default('easy'),
+	 maxTime: z.number().default(5).describe('Number of seconds max time limit default to 5 minutes')
+  }),
+})