@axlsdk/studio 0.9.1 → 0.10.1

package/README.md

@@ -163,36 +163,254 @@ Single endpoint at `ws://localhost:4400/ws` with channel multiplexing:
 
 Channels: `execution:{id}`, `trace:{id}`, `trace:*`, `costs`, `decisions`.
 
+## Embeddable Middleware
+
+For applications using dependency injection (NestJS, etc.) or existing HTTP servers, Studio can be mounted as middleware instead of running as a standalone CLI.
+
+```typescript
+import express from 'express';
+import { AxlRuntime } from '@axlsdk/axl';
+import { createStudioMiddleware } from '@axlsdk/studio/middleware';
+
+const runtime = new AxlRuntime({ providers: ['openai'] });
+// ... register workflows, agents, tools ...
+
+const studio = createStudioMiddleware({
+  runtime,
+  basePath: '/studio',
+  // Your auth logic here — check a token, cookie, or header.
+  // This runs on WebSocket upgrades, which bypass Express middleware.
+  verifyUpgrade: (req) => {
+    const url = new URL(req.url!, `http://${req.headers.host}`);
+    return url.searchParams.get('token') === process.env.MY_SECRET;
+  },
+});
+
+const app = express();
+app.use('/studio', studio.handler);
+
+const server = app.listen(3000);
+studio.upgradeWebSocket(server);
+```
+
+### Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `runtime` | `AxlRuntime` | required | The runtime instance to observe and control |
+| `basePath` | `string` | `''` | URL path prefix (e.g., `'/studio'`) |
+| `serveClient` | `boolean` | `true` | Serve the pre-built SPA |
+| `verifyUpgrade` | `(req) => boolean \| Promise<boolean>` | — | Auth callback for WebSocket upgrades |
+| `readOnly` | `boolean` | `false` | Disable all mutating endpoints |
+| `evals` | `string \| string[] \| { files, conditions? }` | — | Lazy-load eval files for the Eval Runner panel |
+
+### Return value
+
+| Property | Description |
+|----------|-------------|
+| `handler` | Node.js `(req, res)` handler for Express/Fastify/Koa/raw HTTP |
+| `handleWebSocket(ws)` | Handle an individual WebSocket (framework-agnostic) |
+| `upgradeWebSocket(server)` | Attach WS upgrade handling to an `http.Server` |
+| `app` | Underlying Hono app (for Hono-in-Hono mounting) |
+| `connectionManager` | WS connection/channel manager |
+| `close()` | Shut down middleware (removes listeners, closes connections) |
+
+**Note:** `upgradeWebSocket(server)` is required for real-time features (trace streaming, cost updates, execution events, decision resolution). Without it, the Studio SPA loads but panels relying on live data will show no updates. If your framework manages WebSocket connections itself (NestJS gateway, Fastify plugin), use `handleWebSocket()` instead.
+
+### Framework examples
+
+#### NestJS
+
+```typescript
+import { Module, OnModuleInit, OnModuleDestroy } from '@nestjs/common';
+import { HttpAdapterHost } from '@nestjs/core';
+import { createStudioMiddleware, type StudioMiddleware } from '@axlsdk/studio/middleware';
+
+@Module({ /* ... */ })
+export class AppModule implements OnModuleInit, OnModuleDestroy {
+  private studio!: StudioMiddleware;
+
+  constructor(
+    private readonly httpAdapterHost: HttpAdapterHost,
+    private readonly runtime: AxlRuntime, // injected via custom provider
+  ) {}
+
+  onModuleInit() {
+    this.studio = createStudioMiddleware({
+      runtime: this.runtime,
+      basePath: '/studio',
+      verifyUpgrade: (req) => req.headers['authorization'] === `Bearer ${process.env.MY_SECRET}`,
+    });
+
+    // Mount on the underlying Express instance — this is the recommended
+    // NestJS pattern for sub-application mounting (see NestJS HTTP adapter docs).
+    const expressApp = this.httpAdapterHost.httpAdapter.getInstance();
+    expressApp.use('/studio', this.studio.handler);
+    this.studio.upgradeWebSocket(this.httpAdapterHost.httpAdapter.getHttpServer());
+  }
+
+  onModuleDestroy() {
+    this.studio.close();
+  }
+}
+```
+
+#### Fastify
+
+```typescript
+import Fastify from 'fastify';
+import middie from '@fastify/middie';
+import { createStudioMiddleware } from '@axlsdk/studio/middleware';
+
+const studio = createStudioMiddleware({ runtime, basePath: '/studio' });
+const fastify = Fastify();
+
+await fastify.register(middie);
+fastify.use('/studio', studio.handler);
+
+await fastify.listen({ port: 3000 });
+studio.upgradeWebSocket(fastify.server);
+```
+
+#### Raw Node.js
+
+```typescript
+import { createServer } from 'node:http';
+import { createStudioMiddleware } from '@axlsdk/studio/middleware';
+
+const studio = createStudioMiddleware({ runtime });
+const server = createServer(studio.handler);
+studio.upgradeWebSocket(server);
+server.listen(3000);
+```
+
+#### Hono-in-Hono
+
+```typescript
+import { Hono } from 'hono';
+import { createStudioMiddleware, handleWsMessage } from '@axlsdk/studio/middleware';
+
+const studio = createStudioMiddleware({ runtime, basePath: '/studio' });
+const app = new Hono();
+app.route('/studio', studio.app);
+// Wire WebSocket via Hono's native WS support — see spec for full example
+```
+
+### Important: `basePath` must match your mount path
+
+`basePath` tells the SPA where it's mounted in the browser URL. It must match the path in your framework's mount call:
+
+```typescript
+// These must match:
+createStudioMiddleware({ basePath: '/studio' })  // tells the SPA
+app.use('/studio', studio.handler)                // tells Express
+```
+
+If they don't match, the SPA will load but API calls will fail (the SPA sends requests to the wrong path).
+
+### Lazy eval loading
+
+In monorepos, eval files often import from domain modules (prompt builders, validators, fixture datasets) that would create circular dependencies if statically imported from the module that owns the runtime. The `evals` option solves this by dynamically importing eval files on first access to the Eval Runner panel — never during normal API operation.
+
+```typescript
+const studio = createStudioMiddleware({
+  runtime,
+  basePath: '/studio',
+  evals: 'evals/**/*.eval.ts',
+});
+```
+
+Eval files are standalone entry points (like `axl.config.ts`). They can import from any module without creating circular deps in the static module graph, and `@axlsdk/eval` can remain a `devDependency` since bundlers can't see dynamic `import()` calls.
+
+**Multiple patterns or explicit paths:**
+
+```typescript
+evals: ['evals/*.eval.ts', 'tests/evals/*.eval.ts']
+```
+
+**Monorepo import conditions** (process-wide via `module.register()`):
+
+```typescript
+evals: {
+  files: 'libs/api/evals/*.eval.ts',
+  conditions: ['development'],
+}
+```
+
+Each file should `export default` a config with `{ workflow, dataset, scorers }` (the result of `defineEval()`). By default, the runtime executes the named workflow for each dataset item. For self-contained evals that don't depend on a registered workflow, export an `executeWorkflow` function — it will be called instead of `runtime.execute()`. See the [`@axlsdk/eval` README](../axl-eval/README.md#defineevalconfig) for details.
+
+Eval names are the file's path relative to the project root (`cwd`), minus the `.eval.*` suffix:
+
+```
+evals/suggestions.eval.ts        → "evals/suggestions"
+evals/api/accuracy.eval.ts       → "evals/api/accuracy"
+libs/search/accuracy.eval.ts     → "libs/search/accuracy"
+```
+
+This makes names completely stable — a file's name never changes regardless of what other files or patterns exist. You can look at a file path and know its eval name.
+
+Lazy-loaded evals coexist with evals registered directly via `runtime.registerEval()`.
+
+**Important notes:**
+
+- **Caching**: Eval files are loaded once on first access and cached for the lifetime of the middleware. Changes to eval files require a server restart to take effect (both the loader and Node.js module cache are one-shot).
+- **Running nested evals**: Names containing `/` must be URL-encoded in the run endpoint: `POST /api/evals/api%2Faccuracy/run`.
+- **Name stability**: Names are project-relative paths, so they never change when other files or patterns are added/removed.
+- **Supported glob patterns**: `dir/*.eval.ts` (single directory), `dir/**/*.eval.ts` (recursive), `**/*.eval.ts` (recursive from cwd). Multi-segment `**` (e.g., `a/**/b/**/*.ts`) is not supported.
+
+### Security
+
+- **Always** provide `verifyUpgrade` — WebSocket upgrades bypass Express/Fastify/Koa middleware, so your auth middleware does NOT protect WebSocket connections
+- Consider `readOnly: true` for production monitoring — view traces, costs, and schemas without execution capability
+- CORS is not applied in embedded mode — the host framework owns CORS policy
+- `basePath` is validated against unsafe characters and path traversal
+
+### Migrating from the standalone CLI
+
+If you currently use `npx @axlsdk/studio` with a config file:
+
+1. Move runtime creation from `axl.config.ts` into your app's initialization code
+2. Register workflows, agents, and tools on the runtime where they have access to your services
+3. Call `createStudioMiddleware({ runtime, basePath: '/studio' })` and mount the handler
+4. Call `upgradeWebSocket(server)` for WebSocket support
+5. Remove the `axl-studio` CLI from your dev scripts
+
+The `axl.config.ts` file is no longer needed. The standalone CLI continues to work for projects that don't need embedded middleware.
+
 ## Architecture
 
 ```
 src/
   cli.ts                  CLI entry — loads config, starts server
+  middleware.ts           Embeddable middleware: createStudioMiddleware()
   resolve-runtime.ts      Config module interop (ESM default, CJS wrapping, named exports)
   server/
-    index.ts              createServer() — Hono app composition
+    index.ts              createServer() — Hono app composition (basePath, readOnly, cors)
     types.ts              API types, WebSocket message types
     cost-aggregator.ts    Accumulates cost from trace events
     middleware/
       error-handler.ts    Axl errors → JSON error envelope
     routes/               One file per resource (health, workflows, agents, tools, etc.)
     ws/
-      handler.ts          WebSocket message routing
-      connection-manager.ts  Channel subscriptions + broadcast
+      handler.ts          WebSocket message routing (Hono adapter)
+      connection-manager.ts  Channel subscriptions + broadcast (BroadcastTarget)
+      protocol.ts         Shared WS protocol: handleWsMessage(), channel validation
   client/
     App.tsx               React SPA — sidebar + 8 panel routes
     lib/
-      api.ts              Typed fetch wrappers for all endpoints
-      ws.ts               WebSocket client with channel subscriptions
+      api.ts              Typed fetch wrappers (reads window.__AXL_STUDIO_BASE__)
+      ws.ts               WebSocket client with channel subscriptions (reads base path)
     panels/               One directory per panel
 ```
 
-**Server:** Hono HTTP server wrapping the user's `AxlRuntime`. REST endpoints for CRUD, WebSocket for live streaming.
+**Server:** Hono HTTP server wrapping the user's `AxlRuntime`. REST endpoints for CRUD, WebSocket for live streaming. Supports standalone CLI and embeddable middleware modes.
 
-**Client:** React 19 SPA with Tailwind CSS v4, TanStack Query, and react-router-dom. Pre-built at publish time and served as static assets.
+**Client:** React 19 SPA with Tailwind CSS v4, TanStack Query, and react-router-dom. Pre-built at publish time and served as static assets. Reads `window.__AXL_STUDIO_BASE__` for runtime base path configuration.
 
 **CLI:** Auto-detects and loads the user's config via `tsx` (with ESM-forcing resolve hook for `.ts` files), validates the runtime, starts the server, and optionally opens the browser.
 
+**Middleware:** `createStudioMiddleware()` wraps the Hono app as a Node.js `(req, res)` handler via `@hono/node-server`. Adds `verifyUpgrade` for WS auth, `readOnly` mode, and `basePath` injection into the SPA.
+
 ## Development
 
 ```bash