@soederpop/luca 0.0.6 → 0.0.7

package/docs/reports/console-hmr-design.md

@@ -0,0 +1,170 @@
+---
+tags:
+  - console
+  - hmr
+  - repl
+  - design
+---
+# Console HMR Design Research
+
+Add a `--hmr` flag to `luca console` that watches source files and hot-swaps feature instances in the live REPL session, preserving state where possible.
+
+## How the Console Works Today
+
+The `luca console` command (`src/commands/console.ts`) creates a REPL that:
+
+1. Calls `container.helpers.discoverAll()` to load all features, commands, endpoints
+2. Snapshots every available feature into a `featureContext` object via `container.feature(name)` for each name
+3. Optionally loads a `luca.console.ts` project module and merges its exports
+4. Optionally runs `--eval` code/script/markdown before the REPL starts
+5. Creates a `Repl` feature instance with a `vm.Context` built from the snapshot
+6. Enters a hand-rolled readline loop (`repl.ts`) that evaluates expressions in that VM context
+
+The REPL's `_vmContext` is a **mutable plain object** — variables can be reassigned at runtime (`ctx.featureName = newInstance`). Tab completion reads from `Object.keys(ctx)` dynamically, so new/replaced bindings are immediately visible.
+
+## Architectural Facts Relevant to HMR
+
+### helperCache is module-private
+
+`container.ts:618` — `const helperCache = new Map()`. There is no public API to evict or replace a cached feature instance. Calling `container.feature('fs')` with the same options always returns the same object. Any HMR implementation needs either:
+- A new `container.evictHelper(cacheKey)` method
+- A bypass that creates instances outside the cache
+
+### attachToContainer uses configurable: true
+
+`feature.ts` — `Object.defineProperty(this.container, shortcutName, { get: () => this, configurable: true })`. The property descriptor **can** be redefined, which is the only existing affordance for swapping a feature on the container object.
+
+### vm.loadModule() always reads fresh from disk
+
+`vm.ts` — reads file content via `container.fs.readFile()`, transpiles with esbuild `transformSync`, runs in a `vm.Script` context. No module cache to bust. But this runs in a CJS-like VM sandbox — real ES `import` statements inside the file won't work.
+
+### Bun import() cache busting
+
+The only cache-busting pattern in the codebase is `Endpoint.reload()` (`endpoint.ts:176`): `import(\`${path}?t=${Date.now()}\`)`. This works for Bun's native module loader and preserves real ES module semantics.
+
+### State has no serialize/deserialize
+
+`State` (`state.ts`) is an in-memory observable key-value bag with `set()`, `setState()`, `clear()`, and observer callbacks. There is no `toJSON()`/`fromSnapshot()` API. Transferring state between instances requires manually reading `state.current` from old and calling `state.setState()` on new.
+
+### FileManager has chokidar file watching
+
+`file-manager.ts` — `fileManager.watch()` uses chokidar and emits `"file:change"` events with `{ type, path }`. Currently not wired to anything automatically. This is the primitive we'd compose for watching source files.
+
+### Feature self-registration is a static side effect
+
+Features register via `static { Feature.register(this, 'name') }` which stores the **class constructor** in a module-level `FeaturesRegistry` Map. Re-importing a module would call `register()` again with a new constructor — the registry would need to handle overwrites.
+
+## The HMR Flow (Conceptual)
+
+```
+[file change detected]
+  → identify which feature(s) the file maps to
+  → re-import the module (cache-busted)
+  → new class constructor registers over old one
+  → snapshot old instance state: state.current + any serializable instance data
+  → evict old instance from helperCache
+  → create new instance via container.feature(name)
+  → transfer state: newInstance.state.setState(oldState)
+  → patch REPL vm context: ctx[featureName] = newInstance
+  → re-define container shortcut property to point to new instance
+  → print "[HMR] Reloaded: featureName" in REPL
+```
+
+## Open Design Questions
+
+### 1. Scope — which files trigger a reload?
+
+- Just feature source files (`src/node/features/*.ts`)?
+- Also command files, `luca.console.ts`, endpoint files?
+- Or anything under `src/`?
+
+Recommendation: Start with feature files only. Commands and endpoints are less stateful and easier to add later.
+
+### 2. State transfer strategy
+
+- **Best-effort `state.current` transfer**: Read `oldInstance.state.current`, call `newInstance.state.setState(snapshot)`. Simple, covers most cases.
+- **Opt-in hooks**: Features declare `serialize()` / `deserialize()` methods for fine-grained control (e.g., FileManager could note which directories it was watching but not try to transfer the chokidar FSWatcher handle).
+- **Hybrid**: Always transfer `state.current`, and if the feature has a `hmrSerialize()` hook, use that for additional instance data.
+
+Non-state instance data (open file handles, chokidar watchers, readline interfaces, cached esbuild services) cannot be naively transferred. Features with complex resources would need explicit HMR support or accept that those resources restart fresh.
+
+### 3. Module re-import strategy
+
+Two options:
+
+**Option A — Bun `import()` with `?t=` cache busting**: Real ES module semantics, `import` statements inside the feature file work. The new module's `static {}` block re-registers the class. This is what `Endpoint.reload()` already does.
+
+**Option B — `vm.loadModule()`**: Always reads fresh from disk, no cache issues. But runs in a CJS sandbox — internal `import` statements won't resolve. Feature files heavily use `import`, so this likely won't work.
+
+Recommendation: Option A. It's proven in the codebase and preserves full module semantics.
+
+### 4. Registry re-registration
+
+`Feature.register()` currently does `features.register(id, SubClass)` which calls `registry.members.set(id, SubClass)`. A re-import with a new class constructor would overwrite the old entry. This actually works — `Map.set` overwrites silently. But we should verify there are no side effects in `interceptRegistration` hooks or other registration logic that would break.
+
+### 5. REPL context patching — automatic vs explicit
+
+- **Automatic**: FileManager watches, detects change, swaps feature, patches `ctx[name]`, prints HMR message. User sees updated behavior on next expression.
+- **Explicit**: User types `hmr.reload('fs')` or similar in the REPL to trigger a reload manually.
+- **Both**: Auto-reload on file change, plus a manual `hmr.reload('name')` for forcing reloads or reloading things that aren't file-backed.
+
+Recommendation: Both. Auto is the main UX, manual is the escape hatch.
+
+### 6. Feedback in the REPL
+
+Print a colored message when a feature reloads:
+```
+[HMR] Reloaded: fs (state transferred)
+[HMR] Reloaded: diskCache (fresh — no prior state)
+[HMR] Error reloading vm: SyntaxError: Unexpected token (kept old instance)
+```
+
+This should be non-intrusive — printed above the prompt line if possible.
+
+### 7. Failure mode
+
+If the new code has a syntax error or the constructor throws:
+- Keep the old instance alive
+- Print the error in the REPL
+- Do not crash the session
+
+This is the only sane approach for a dev tool.
+
+## Implementation Sketch
+
+### New infrastructure needed
+
+1. **`container.evictHelper(type, id, options?)`** — public method on Container that deletes from `helperCache` and cleans up `featureIdToHelperCacheKeyMap` and `contextMap`
+2. **`Feature.prototype.hmrSerialize?()` / `hmrDeserialize?(data)`** — optional hooks for features that need custom state transfer beyond `state.current`
+3. **`registry.register()` handling overwrites** — verify this works cleanly, add a `"re-registered"` event if useful
+4. **File-to-feature mapping** — a way to know that `src/node/features/disk-cache.ts` corresponds to the `diskCache` feature ID
+
+### Changes to existing code
+
+1. **`src/commands/console.ts`** — add `--hmr` flag to `argsSchema`, wire up file watching and the reload loop when enabled
+2. **`src/container.ts`** — expose `evictHelper()` (or a more targeted `replaceFeature()`)
+3. **`src/node/features/repl.ts`** — expose a method to patch the VM context (or just expose `_vmContext` which is already accessible)
+
+### Rough dependency graph
+
+```
+argsSchema adds --hmr flag
+  → console handler checks for --hmr
+  → starts FileManager.watch() on src/ directory
+  → subscribes to "file:change" events
+  → on change: resolveFeatureFromPath(changedFile)
+  → cache-bust import the module
+  → container.evictHelper('feature', featureId)
+  → newInstance = container.feature(featureId, { enable: wasEnabled })
+  → transfer state from old → new
+  → patch repl._vmContext[featureId] = newInstance
+  → print HMR message
+```
+
+## Risks and Unknowns
+
+- **Circular dependency during re-import**: If feature A imports feature B at module level, and both are being reloaded, the order matters. May need to batch reloads or do a dependency-aware reload order.
+- **Event listener cleanup**: Old feature instances may have registered listeners on the container event bus. Need to remove those or they'll fire on stale instances.
+- **Observer cleanup**: State observers from the old instance need to be unsubscribed or they'll leak.
+- **Features that modify globals**: Some features might set up global state (process event handlers, etc.) that won't be cleaned up by replacing the instance.
+- **The `?t=` trick and TypeScript**: Bun handles `import('./foo.ts?t=123')` but we should verify this works for all feature files, especially those with complex re-exports.

package/docs/reports/helper-semantic-search.md

@@ -0,0 +1,72 @@
+---
+tags:
+  - feature-design
+  - semantic-search
+  - introspection
+status: draft
+---
+# Helper Semantic Search Feature
+
+Build semantic search over all describable luca helpers so AI assistants can find the right feature/client/server by describing what they need.
+
+## Motivation
+
+The `luca describe` system produces rich markdown for every helper (features, clients, servers, commands, endpoints, selectors). An AI assistant working with luca currently has to know the exact name of a helper to look it up. Semantic search would let it say "I need to run shell commands" and find `proc`, or "I need to cache things to disk" and find `diskCache`.
+
+Storage location: `~/.luca/embeddings/`
+
+## What Already Exists
+
+Two systems that combine perfectly:
+
+1. **SemanticSearch feature** (`src/node/features/semantic-search.ts`) — SQLite-backed embedding engine with OpenAI/local GGUF providers, section-based chunking, BM25 + vector + hybrid search with RRF fusion.
+
+2. **Introspection system** (`src/introspection/`) — Every helper produces structured `HelperIntrospection` JSON and rendered markdown via `Helper.introspectAsText()`. Build-time AST scanning (JSDoc) + runtime Zod schema reflection. The `__INTROSPECTION__` map holds everything.
+
+## Proposed Design
+
+### Approach
+
+Reuse the existing `SemanticSearch` feature directly. Create a new feature (e.g. `helperSearch`) that:
+
+1. Iterates all registries, calls `introspectAsText()` on each helper to get markdown
+2. Structures the markdown as `DocumentInput` objects with sections (methods, getters, events, state, options)
+3. Feeds them to SemanticSearch for embedding and indexing
+4. Exposes a `search(query)` method that delegates to hybridSearch
+
+### Storage
+
+DB stored at `~/.luca/embeddings/helpers.<provider>-<model>.sqlite`, scoped by provider+model like contentbase does.
+
+### When to Build Index
+
+- Lazy on first search if no index exists or if stale
+- Explicit rebuild via `luca search --rebuild`
+- Content hash gating from SemanticSearch handles incremental updates automatically
+
+### CLI Surface
+
+`luca search "file operations"` — returns ranked helpers with snippets showing why they matched.
+
+### MCP / AI Assistant Surface
+
+Expose as a tool in the luca-sandbox MCP so AI assistants can search for helpers by describing what they need.
+
+## Open Questions
+
+1. **Scope** — Index just core luca helpers, or also project-level commands/endpoints/selectors discovered at runtime? Suggestion: core always, project-level optionally.
+
+2. **Granularity** — One document per helper (full describe output) vs chunked by section (methods, events, state). Suggestion: chunk by section so "run shell commands" matches `proc.exec` specifically.
+
+3. **Primary consumer** — MCP tool for AI assistants, CLI for humans, or both? Suggestion: both.
+
+4. **Embedding provider default** — OpenAI (higher quality, needs API key) or local GGUF (works offline, lower quality)? Suggestion: OpenAI default with local fallback.
+
+## Key Files
+
+- `src/node/features/semantic-search.ts` — The embedding engine to reuse
+- `src/introspection/index.ts` — `__INTROSPECTION__` map, `HelperIntrospection` type
+- `src/helper.ts` — `Helper.introspect()`, `Helper.introspectAsText()`, markdown renderers
+- `src/registry.ts` — Registry base class, `describe()`, `describeAll()`
+- `src/commands/describe.ts` — Current describe command (target resolution, rendering)
+- `src/node/features/content-db.ts` — Reference for how contentDb wraps SemanticSearch

package/docs/scaffolds/client.md

@@ -11,9 +11,8 @@ When to build a client:
 
 ```ts
 import { z } from 'zod'
-import { Client, clients, RestClient } from '@soederpop/luca/client'
+import { Client, RestClient } from '@soederpop/luca/client'
 import { ClientStateSchema, ClientOptionsSchema, ClientEventsSchema } from '@soederpop/luca'
-import type { ContainerContext } from '@soederpop/luca'
 ```
 
 Use `RestClient` for HTTP APIs (most common). It gives you `get`, `post`, `put`, `patch`, `delete` methods that handle JSON, headers, and error wrapping.
@@ -36,6 +35,8 @@ export type {{PascalName}}Options = z.infer<typeof {{PascalName}}OptionsSchema>
 
 ## Class
 
+Running `luca introspect` captures JSDoc blocks and Zod schemas and includes them in the description whenever somebody calls `container.clients.describe('{{camelName}}')` or `luca describe {{camelName}}`.
+
 ```ts
 /**
  * {{description}}
@@ -51,14 +52,14 @@ export class {{PascalName}} extends RestClient<{{PascalName}}State, {{PascalName
   static override shortcut = 'clients.{{camelName}}' as const
   static override stateSchema = {{PascalName}}StateSchema
   static override optionsSchema = {{PascalName}}OptionsSchema
-  static override description = '{{description}}'
-
-  constructor(options: {{PascalName}}Options, context: ContainerContext) {
-    options = {
-      ...options,
-      baseURL: options.baseURL || 'https://api.example.com',
-    }
-    super(options, context)
+  static { Client.register(this, '{{camelName}}') }
+
+  /**
+   * Called after the client is initialized. Use this for any setup logic
+   * instead of overriding the constructor.
+   */
+  async afterInitialize() {
+    // Set up default headers, configure auth, etc.
   }
 
   // Add API methods here. Each wraps an endpoint.
@@ -69,6 +70,8 @@ export class {{PascalName}} extends RestClient<{{PascalName}}State, {{PascalName
 }
 ```
 
+**Important**: You almost never need to override the constructor. Use `afterInitialize()` for setup logic — it runs after the client is fully wired into the container. Set `baseURL` via the options schema default instead of constructor manipulation.
+
 ## Module Augmentation
 
 ```ts
@@ -81,17 +84,22 @@ declare module '@soederpop/luca/client' {
 
 ## Registration
 
+Registration happens inside the class body using a static block. The default export is just the class itself.
+
 ```ts
-export default clients.register('{{camelName}}', {{PascalName}})
+// Inside the class:
+static { Client.register(this, '{{camelName}}') }
+
+// At module level:
+export default {{PascalName}}
 ```
 
 ## Complete Example
 
 ```ts
 import { z } from 'zod'
-import { clients, RestClient } from '@soederpop/luca/client'
+import { Client, RestClient } from '@soederpop/luca/client'
 import { ClientStateSchema, ClientOptionsSchema } from '@soederpop/luca'
-import type { ContainerContext } from '@soederpop/luca'
 
 declare module '@soederpop/luca/client' {
   interface AvailableClients {
@@ -121,20 +129,21 @@ export class {{PascalName}} extends RestClient<{{PascalName}}State, {{PascalName
   static override shortcut = 'clients.{{camelName}}' as const
   static override stateSchema = {{PascalName}}StateSchema
   static override optionsSchema = {{PascalName}}OptionsSchema
-  static override description = '{{description}}'
+  static { Client.register(this, '{{camelName}}') }
 
-  constructor(options: {{PascalName}}Options, context: ContainerContext) {
-    super({ ...options, baseURL: options.baseURL }, context)
+  async afterInitialize() {
+    // Setup logic goes here — not in the constructor
   }
 }
 
-export default clients.register('{{camelName}}', {{PascalName}})
+export default {{PascalName}}
 ```
 
 ## Conventions
 
 - **Extend RestClient for HTTP**: It gives you typed HTTP methods. Only use base `Client` if you need a non-HTTP protocol.
-- **Set baseURL in constructor**: Override options to hardcode or default the API base URL.
+- **Set baseURL via options schema**: Use a Zod `.default()` on the `baseURL` field rather than overriding the constructor.
+- **Use `afterInitialize()`**: For any setup logic (auth, default headers, etc.) instead of overriding the constructor.
 - **Wrap endpoints as methods**: Each API endpoint gets a method. Keep them thin — just map to HTTP calls.
-- **JSDoc everything**: Every public method needs `@param`, `@returns`, `@example`.
-- **Auth in options**: Pass API keys, tokens via options schema. Check them in the constructor or a setup method.
+- **JSDoc everything**: Every public method needs `@param`, `@returns`, `@example`. Run `luca introspect` after changes to update generated docs.
+- **Auth in options**: Pass API keys, tokens via options schema. Check them in `afterInitialize()` or a setup method.

package/docs/scaffolds/command.md

@@ -1,6 +1,6 @@
 # Building a Command
 
-A command extends the `luca` CLI. Commands live in a project's `commands/` folder and are automatically discovered. They receive parsed options and a container context.
+A command extends the `luca` CLI. Commands live in a project's `commands/` folder and are automatically discovered. They are Helper subclasses under the hood — the framework grafts your module exports into a Command class at runtime.
 
 When to build a command:
 - You need a CLI task for a project (build scripts, generators, automation)
@@ -11,60 +11,54 @@ When to build a command:
 
 ```ts
 import { z } from 'zod'
-import { commands, CommandOptionsSchema } from '@soederpop/luca'
 import type { ContainerContext } from '@soederpop/luca'
 ```
 
-## Args Schema
+## Positional Arguments
 
-Define your command's arguments and flags. Extend `CommandOptionsSchema` which gives you `_` (positional args) and `name` for free.
+Export a `positionals` array to map CLI positional args into named options fields. The first positional (`_[0]`) is always the command name — `positionals` maps `_[1]`, `_[2]`, etc.
 
 ```ts
-export const argsSchema = CommandOptionsSchema.extend({
-  // Add your flags here. Each becomes a --flag on the CLI.
-  // Example: verbose: z.boolean().default(false).describe('Enable verbose output'),
-  // Example: output: z.string().optional().describe('Output file path'),
-})
+// luca {{kebabName}} ./src  =>  options.target === './src'
+export const positionals = ['target']
 ```
 
-## Handler
+## Args Schema
 
-The handler function receives parsed options and the container context. Use the container for all I/O.
+Define your command's arguments and flags with Zod. Each field becomes a `--flag` on the CLI. Fields named in `positionals` also accept positional args.
 
 ```ts
-export default async function {{camelName}}(options: z.infer<typeof argsSchema>, context: ContainerContext) {
-  const container = context.container as any
-  const fs = container.feature('fs')
-  const args = container.argv._ as string[]
+export const argsSchema = z.object({
+  // Positional: first arg after command name (via positionals array above)
+  // target: z.string().optional().describe('The target to operate on'),
 
-  // args[0] is your command name, args[1+] are positional arguments
-  // options contains parsed --flags
-
-  // Your implementation here
-}
+  // Flags: passed as --flag on the CLI
+  // verbose: z.boolean().default(false).describe('Enable verbose output'),
+  // output: z.string().optional().describe('Output file path'),
+})
 ```
 
-## Registration
+## Description
 
-Register the command at the bottom of the file. The `description` shows up in `luca --help`.
+Export a description string for `luca --help` display:
 
 ```ts
-commands.registerHandler('{{camelName}}', {
-  description: '{{description}}',
-  argsSchema,
-  handler: {{camelName}},
-})
+export const description = '{{description}}'
 ```
 
-## Module Augmentation
+## Handler
 
-Optional but gives TypeScript autocomplete for `commands.lookup('yourCommand')`.
+Export a default async function. It receives parsed options and the container context. Use the container for all I/O. Positional args declared in the `positionals` export are available as named fields on `options`.
 
 ```ts
-declare module '@soederpop/luca' {
-  interface AvailableCommands {
-    {{camelName}}: ReturnType<typeof commands.registerHandler>
-  }
+export default async function {{camelName}}(options: z.infer<typeof argsSchema>, context: ContainerContext) {
+  const { container } = context
+  const fs = container.feature('fs')
+
+  // options.target is set from the first positional arg (via positionals export)
+  // options.verbose, options.output, etc. come from --flags
+
+  // Your implementation here
 }
 ```
 
@@ -72,35 +66,55 @@ declare module '@soederpop/luca' {
 
 ```ts
 import { z } from 'zod'
-import { commands, CommandOptionsSchema } from '@soederpop/luca'
 import type { ContainerContext } from '@soederpop/luca'
 
-declare module '@soederpop/luca' {
-  interface AvailableCommands {
-    {{camelName}}: ReturnType<typeof commands.registerHandler>
-  }
-}
+export const description = '{{description}}'
 
-export const argsSchema = CommandOptionsSchema.extend({})
+// Map positional args to named options: luca {{kebabName}} myTarget => options.target === 'myTarget'
+export const positionals = ['target']
+
+export const argsSchema = z.object({
+  target: z.string().optional().describe('The target to operate on'),
+})
 
 export default async function {{camelName}}(options: z.infer<typeof argsSchema>, context: ContainerContext) {
-  const container = context.container as any
+  const { container } = context
   const fs = container.feature('fs')
 
-  console.log('{{camelName}} running...')
+  console.log('{{kebabName}} running...', options.target)
 }
+```
 
-commands.registerHandler('{{camelName}}', {
-  description: '{{description}}',
-  argsSchema,
-  handler: {{camelName}},
-})
+## Container Properties
+
+The `context.container` object provides useful properties beyond features:
+
+```ts
+export default async function {{camelName}}(options: z.infer<typeof argsSchema>, context: ContainerContext) {
+  const { container } = context
+
+  // Current working directory
+  container.cwd                    // '/path/to/project'
+
+  // Path utilities (scoped to cwd)
+  container.paths.resolve('src')   // '/path/to/project/src'
+  container.paths.join('a', 'b')   // '/path/to/project/a/b'
+  container.paths.relative('src')  // 'src'
+
+  // Package manifest (parsed package.json)
+  container.manifest.name          // 'my-project'
+  container.manifest.version       // '1.0.0'
+
+  // Raw CLI arguments (from minimist) — prefer positionals export for positional args
+  container.argv                   // { _: ['{{kebabName}}', ...], verbose: true, ... }
+}
 ```
 
 ## Conventions
 
-- **File location**: `commands/{{camelName}}.ts` in the project root. The `luca` CLI discovers these automatically.
-- **Naming**: camelCase for both file and registration ID. `luca my-command` maps to `commands/my-command.ts`.
+- **File location**: `commands/{{kebabName}}.ts` in the project root. The `luca` CLI discovers these automatically.
+- **Naming**: kebab-case for filename. `luca {{kebabName}}` maps to `commands/{{kebabName}}.ts`.
 - **Use the container**: Never import `fs`, `path`, `child_process` directly. Use `container.feature('fs')`, `container.paths`, `container.feature('proc')`.
-- **Positional args**: Access via `container.argv._` — it's an array where `_[0]` is the command name.
+- **Positional args**: Export `positionals = ['name1', 'name2']` to map CLI positional args into named options fields. For raw access, use `container.argv._` where `_[0]` is the command name.
 - **Exit codes**: Return nothing for success. Throw for errors — the CLI catches and reports them.
+- **Help text**: Use `.describe()` on every schema field — it powers `luca {{kebabName}} --help`.