@auto-engineer/cli 1.148.0 → 1.150.0

package/CHANGELOG.md

@@ -1,5 +1,47 @@
 # @auto-engineer/cli
 
+## 1.150.0
+
+### Minor Changes
+
+- [`a502d47`](https://github.com/BeOnAuto/auto-engineer/commit/a502d472df528a1c5d53905898220c2a3a49d8a5) Thanks [@osamanar](https://github.com/osamanar)! -
+
+- [`c96a70e`](https://github.com/BeOnAuto/auto-engineer/commit/c96a70e49f9f64213450d0b9840bc9a8b2b1d77d) Thanks [@osamanar](https://github.com/osamanar)! -
+
+- [`4343bca`](https://github.com/BeOnAuto/auto-engineer/commit/4343bcaa2576703ae578fd9b7f5ec5b9776702a9) Thanks [@osamanar](https://github.com/osamanar)! -
+
+- [`28ffce9`](https://github.com/BeOnAuto/auto-engineer/commit/28ffce9c8b95f0c3ca61728bd1f667fa9416d461) Thanks [@osamanar](https://github.com/osamanar)! -
+
+- [`3fd5dbf`](https://github.com/BeOnAuto/auto-engineer/commit/3fd5dbfbfb4f7a5fe71ae53105b31a1b0f30f911) Thanks [@osamanar](https://github.com/osamanar)! -
+
+### Patch Changes
+
+- [`41f3df3`](https://github.com/BeOnAuto/auto-engineer/commit/41f3df3025445ba92208c2b007b8e29a40489309) Thanks [@github-actions[bot]](https://github.com/github-actions%5Bbot%5D)! - - **global**: version packages
+  - **global**: major refresh to all the docs
+- Updated dependencies [[`a502d47`](https://github.com/BeOnAuto/auto-engineer/commit/a502d472df528a1c5d53905898220c2a3a49d8a5), [`c96a70e`](https://github.com/BeOnAuto/auto-engineer/commit/c96a70e49f9f64213450d0b9840bc9a8b2b1d77d), [`4343bca`](https://github.com/BeOnAuto/auto-engineer/commit/4343bcaa2576703ae578fd9b7f5ec5b9776702a9), [`28ffce9`](https://github.com/BeOnAuto/auto-engineer/commit/28ffce9c8b95f0c3ca61728bd1f667fa9416d461), [`41f3df3`](https://github.com/BeOnAuto/auto-engineer/commit/41f3df3025445ba92208c2b007b8e29a40489309), [`3fd5dbf`](https://github.com/BeOnAuto/auto-engineer/commit/3fd5dbfbfb4f7a5fe71ae53105b31a1b0f30f911)]:
+  - @auto-engineer/file-store@1.150.0
+  - @auto-engineer/narrative@1.150.0
+  - @auto-engineer/pipeline@1.150.0
+
+## 1.149.0
+
+### Minor Changes
+
+- [`e1eebbd`](https://github.com/BeOnAuto/auto-engineer/commit/e1eebbdf4f209780e790094d2e6887c4fa809f98) Thanks [@github-actions[bot]](https://github.com/github-actions%5Bbot%5D)! - - **server-generator-apollo-emmett**: add Given state ref hints to state.ts.ejs
+  - **server-generator-apollo-emmett**: context-aware nonCommandField instructions
+  - **server-generator-apollo-emmett**: add state context instruction
+  - **server-generator-apollo-emmett**: extract shared template helpers
+  - **server-generator-apollo-emmett**: filter state refs from hasGivenEvents in decide.ts.ejs
+
+### Patch Changes
+
+- [`d38c81e`](https://github.com/BeOnAuto/auto-engineer/commit/d38c81e7bb442a39626564cf4f6d8d55b60d0a38) Thanks [@SamHatoum](https://github.com/SamHatoum)! -
+
+- Updated dependencies [[`d38c81e`](https://github.com/BeOnAuto/auto-engineer/commit/d38c81e7bb442a39626564cf4f6d8d55b60d0a38), [`e1eebbd`](https://github.com/BeOnAuto/auto-engineer/commit/e1eebbdf4f209780e790094d2e6887c4fa809f98)]:
+  - @auto-engineer/file-store@1.149.0
+  - @auto-engineer/narrative@1.149.0
+  - @auto-engineer/pipeline@1.149.0
+
 ## 1.148.0
 
 ### Minor Changes

package/README.md

@@ -1,35 +1,39 @@
 # @auto-engineer/cli
 
-Command-line interface for Auto Engineer, a tool for building applications with Narrative Driven Development.
-
----
+Command-line interface that starts the Auto Engineer pipeline server, loads plugins, synchronizes local files with remote sandboxes, and optionally exposes a public tunnel.
 
 ## Purpose
 
-The CLI orchestrates a pipeline-based architecture that loads plugins, starts development servers, and synchronizes files between local development and remote sandboxes. It serves as the primary entry point for running Auto Engineer workflows.
+The CLI is the primary entry point for Auto Engineer. It:
 
----
+- Loads an `auto.config.ts` configuration file and registers pipeline plugins and command handlers
+- Starts an HTTP + WebSocket server that exposes pipeline endpoints and real-time file sync
+- Watches the local `narratives/` directory (configurable) and streams file changes to connected clients via Socket.IO
+- Discovers TypeScript type definitions for external dependencies and includes them in the sync payload
+- Optionally opens an ngrok tunnel for remote access from the Auto cloud app
+- Renders a terminal UI (Ink-based) that aggregates stdout/stderr from registered child services
 
 ## Installation
 
 ```bash
-# Global
+# Global install
 npm install -g @auto-engineer/cli
 
-# Or via npx
+# Or run directly
 npx @auto-engineer/cli
 ```
 
+Both `auto` and `auto-engineer` are registered as bin commands.
+
 ## Quick Start
 
 ```bash
-# Step 1: Create a config file
+# 1. Create a config file in your project root
 cat > auto.config.ts << 'EOF'
 import { define } from '@auto-engineer/pipeline';
 
 export const plugins = [
   '@auto-engineer/narrative',
-  '@auto-engineer/server-generator-apollo-emmett',
 ];
 
 export const pipeline = define('my-pipeline')
@@ -41,100 +45,135 @@ export const pipeline = define('my-pipeline')
   .build();
 EOF
 
-# Step 2: Start the server
+# 2. Start the server (launches TUI by default)
 auto start
 
-# Step 3: View the pipeline diagram
+# 3. In another terminal, check status
+auto status
+
+# 4. Dispatch a command
+auto dispatch GenerateServer --data '{"modelPath": "./schema.json", "destination": "."}'
+
+# 5. Open the pipeline diagram in a browser
 auto diagram
 ```
 
----
-
 ## How-to Guides
 
-### Start the Pipeline Server
+### Start with Legacy Console Output
 
 ```bash
-auto start
-# or simply
-auto
+auto start --no-tui
 ```
 
-### Dispatch a Command
+### Connect to a Remote Server
 
 ```bash
-auto dispatch GenerateServer --data '{"modelPath": "./schema.json", "destination": "."}'
+auto status --host http://remote-server:5555
+auto dispatch MyCommand --host http://remote-server:5555 --data '{}'
 ```
 
-### Check Server Status
+### Enable an ngrok Tunnel
+
+Requires a free [ngrok account](https://ngrok.com/signup).
 
 ```bash
-auto status
+# Via CLI flag
+auto start --tunnel
+
+# Or in auto.config.ts
+export const tunnel = {
+  enabled: true,
+  authtoken: 'your-ngrok-authtoken',  // or set NGROK_AUTHTOKEN in .env
+  domain: 'my-custom.ngrok.io',       // optional reserved domain
+};
+```
+
+When the tunnel is active, the CLI generates a bearer token and prints it alongside the public URL. Both are needed to connect from the Auto cloud app.
+
+### Configure File Sync Directory
+
+By default the file syncer watches `narratives/` relative to the config file. Override it:
+
+```typescript
+// auto.config.ts
+export const fileSync = {
+  dir: './custom-sync-dir',
+};
 ```
 
-### Connect to Existing Server
+### Supply AI Provider Credentials
+
+The file syncer encrypts provider credentials into a JWE token and includes it in the synced `auto.config.json`. Set one of these in your `.env`:
 
 ```bash
-auto dispatch MyCommand --host http://localhost:5555 --data '{}'
+ANTHROPIC_API_KEY=sk-...
+OPENAI_API_KEY=sk-...
+GEMINI_API_KEY=...
+XAI_API_KEY=...
+
+# Or a custom provider
+CUSTOM_PROVIDER_NAME=my-provider
+CUSTOM_PROVIDER_BASE_URL=https://api.example.com
+CUSTOM_PROVIDER_API_KEY=sk-...
+CUSTOM_PROVIDER_DEFAULT_MODEL=my-model
+
+# Optional: override the default model for any provider
+DEFAULT_AI_MODEL=claude-sonnet-4-20250514
 ```
 
 ### Configure for CI/CD
 
 ```bash
-auto start --port 8080 --config ./ci.config.ts
+auto start --port 8080 --config ./ci.config.ts --no-tui
 ```
 
----
+The `--no-tui` flag disables the interactive terminal UI, which is required for non-interactive environments.
 
 ## CLI Reference
 
-### Commands
+### Global Options
 
-#### `auto start`
+| Option | Alias | Type | Default | Description |
+|--------|-------|------|---------|-------------|
+| `--port` | `-p` | number | `5555` | Server port (auto-increments if taken, up to +100) |
+| `--debug` | `-d` | boolean | `false` | Enable debug mode |
+| `--config` | `-c` | string | `auto.config.ts` | Path to config file |
+| `--host` | | string | `localhost` | Connect to an existing server |
+| `--tunnel` | | boolean | `false` | Enable ngrok tunnel |
+| `--no-tui` | | boolean | `false` | Disable TUI, use legacy console output |
+| `--version` | `-v` | | | Print version |
 
-Start the pipeline server with loaded config (default command).
+### Commands
 
-```bash
-auto start [options]
-```
+#### `auto start` (default)
 
-| Option | Alias | Type | Default | Description |
-|--------|-------|------|---------|-------------|
-| `--port` | `-p` | number | 5555 | Server port |
-| `--debug` | `-d` | boolean | false | Enable debug mode |
-| `--config` | `-c` | string | auto.config.ts | Path to config file |
+Start the pipeline server. This is the default command -- running `auto` with no subcommand is equivalent.
 
 #### `auto dispatch <command>`
 
-Dispatch a command to the pipeline server.
-
-```bash
-auto dispatch <command> [options]
-```
+Send a command to the running pipeline server.
 
 | Option | Type | Default | Description |
 |--------|------|---------|-------------|
-| `--data` | string | `{}` | Command data as JSON |
-| `--host` | string | localhost | Connect to existing server |
-
-#### `auto status`
-
-Check pipeline server health and registry status.
+| `--data` | string (JSON) | `{}` | Command payload |
 
 ```bash
-auto status
+auto dispatch GenerateServer --data '{"modelPath": "./schema.json"}'
 ```
 
-#### `auto diagram`
+#### `auto status`
 
-Open the pipeline diagram in a browser.
+Query the server's `/health` and `/registry` endpoints. Prints uptime, registered command count, and event handler count.
 
-```bash
-auto diagram
-```
+#### `auto diagram`
+
+Open the pipeline diagram (`/pipeline/diagram`) in the default browser.
 
 ### Configuration File
 
+The CLI looks for `auto.config.ts` or `auto.config.js` in the current working directory.
+
 ```typescript
 // auto.config.ts
 import { define } from '@auto-engineer/pipeline';
@@ -153,89 +192,201 @@ export const pipeline = define('my-pipeline')
     destination: e.data.directory,
   }))
   .build();
+
+// Optional
+export const fileSync = {
+  dir: 'narratives',           // directory to watch and sync
+};
+
+export const tunnel = {
+  enabled: false,              // enable ngrok tunnel
+  authtoken: 'ngrok-token',   // or use NGROK_AUTHTOKEN env var
+  domain: 'my.ngrok.io',      // optional reserved domain
+  token: 'custom-bearer',     // optional fixed bearer token
+};
+
+export const COMMANDS = [
+  {
+    name: 'MyCommand',
+    handle: async (data) => ({ type: 'MyCommandDone', data: {} }),
+  },
+];
 ```
 
----
+### Server Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/health` | GET | Health check (status, uptime) |
+| `/registry` | GET | List registered command and event handlers |
+| `/pipeline` | GET | Pipeline state |
+| `/pipeline/diagram` | GET | Visual pipeline diagram |
+| `/events` | GET | SSE event stream |
+| `/command` | POST | Dispatch a command (`{ type, data }`) |
+| `/configure` | POST | Set service token and org/project IDs |
+| `ws://.../file-sync` | Socket.IO | Bi-directional file sync |
+
+### Subpath Exports
+
+| Export | Description |
+|--------|-------------|
+| `@auto-engineer/cli` | `startServer`, `getServiceRegistry`, types |
+| `@auto-engineer/cli/server` | Direct server module import |
 
 ## Troubleshooting
 
-### Server Won't Start
+### "No pipeline config found" on Start
 
-**Symptom:** `No pipeline config found` error
+**Symptom:** Server exits with `No pipeline config found. Create an auto.config.ts file.`
 
-**Cause:** Missing auto.config.ts file in current directory
+**Cause:** No `auto.config.ts` or `auto.config.js` exists in the current working directory.
 
 **Solution:**
 
 ```bash
-# Create a minimal config
 cat > auto.config.ts << 'EOF'
+import { define } from '@auto-engineer/pipeline';
 export const plugins = [];
-export const pipeline = { nodes: [] };
+export const pipeline = define('default').build();
 EOF
 ```
 
 ### Port Already in Use
 
-**Symptom:** Server fails to bind to port
+**Symptom:** Server starts on a different port than expected.
 
-**Cause:** Another process is using the port
+**Cause:** The requested port is occupied. The CLI auto-scans up to 100 ports above the requested one via `get-port`.
+
+**Solution:** Check the actual port in the startup output, or specify a known-free port:
+
+```bash
+auto start --port 6000
+```
+
+### Plugin Command Handlers Not Found
+
+**Symptom:** Dispatching a command returns an error about missing handlers.
+
+**Cause:** The plugin package is not installed, not built, or does not export a `COMMANDS` array.
+
+**Solution:**
+
+1. Install the plugin: `pnpm add @auto-engineer/my-plugin`
+2. Build it: `pnpm --filter @auto-engineer/my-plugin build`
+3. Verify it exports `COMMANDS` from its main entry point or has a `commands/` directory with `.js` files that export handlers
+
+### ngrok Tunnel Fails to Start
+
+**Symptom:** Warning `Failed to start ngrok tunnel` appears; server continues without tunnel.
+
+**Cause:** Missing or invalid ngrok authtoken.
 
 **Solution:**
 
 ```bash
-auto start --port 5556
+# Set in .env
+echo 'NGROK_AUTHTOKEN=your-token' >> .env
+
+# Or in auto.config.ts
+export const tunnel = {
+  enabled: true,
+  authtoken: 'your-token',
+};
 ```
 
-### Plugin Not Loading
+### TUI Not Rendering Correctly
 
-**Symptom:** Command handlers not found
+**Symptom:** Garbled output or blank screen.
 
-**Cause:** Plugin package not installed or COMMANDS not exported
+**Cause:** Terminal does not support the Ink rendering framework, or the process is not attached to a TTY.
 
 **Solution:**
 
 ```bash
-pnpm add @auto-engineer/my-plugin
+auto start --no-tui
 ```
 
 ### Enable Debug Logging
 
 ```bash
+# All auto debug output
 DEBUG=auto:* auto start
-```
 
----
+# File syncer only
+DEBUG=auto:cli:file-syncer auto start
+
+# Type definition resolution
+DEBUG=auto-engineer:file-syncer:* auto start
+```
 
 ## Architecture
 
 ```
+bin/
+  auto.ts                  # Thin shim: spawns src/index.js with forwarded args
 src/
-├── index.ts
-├── file-syncer/
-│   └── index.ts
-└── bin/
-    └── auto.js
+  index.ts                 # CLI entry: Commander program, start/dispatch/status/diagram
+  server.ts                # startServer(): config loading, plugin discovery, PipelineServer + FileSyncer
+  config-loader.ts         # Loads auto.config.ts via jiti, resolves defaults
+  tunnel.ts                # ngrok tunnel, bearer-token auth middleware, service-token store
+  tui/
+    index.tsx              # renderTui() — Ink render entry
+    App.tsx                # Root Ink component
+    Dashboard.tsx          # Dashboard UI
+    ServiceRegistry.ts     # Tracks registered child services (id, streams, process)
+    LogBuffer.ts           # Ring buffer for service log lines
+    hooks/
+      useServiceStreams.ts  # Wires Readable streams into LogBuffer
+  file-syncer/
+    index.ts               # FileSyncer class: chokidar watch, Socket.IO sync, auto.config.json virtual file
+    types/wire.ts           # WireChange, WireInitial protocol types
+    sync/
+      resolveSyncFileSet.ts # Computes the set of files to sync (narratives + .d.ts externals)
+    crypto/
+      jwe-encryptor.ts     # Encrypts AI provider credentials as JWE using JWKS public key
+      provider-resolver.ts # Detects active AI provider from env vars
+    discovery/
+      bareImports.ts       # Extracts bare import specifiers from source files
+      dts.ts               # Probes node_modules for entry .d.ts files
+    utils/
+      hash.ts              # md5, readBase64, statSize via NodeFileStore
+      path.ts              # Wire path encoding/decoding, external path remapping
 ```
 
-### Server Endpoints
+```mermaid
+graph TD
+    A[bin/auto.ts] --> B[src/index.ts<br/>Commander CLI]
+    B --> C[src/server.ts<br/>startServer]
+    C --> D["@auto-engineer/pipeline<br/>PipelineServer"]
+    C --> E[file-syncer/index.ts<br/>FileSyncer]
+    C --> F[config-loader.ts]
+    C --> G[tui/index.tsx]
+    B --> H[tunnel.ts<br/>ngrok + auth]
+    E --> I["@auto-engineer/file-store<br/>NodeFileStore"]
+    E --> J["@auto-engineer/narrative<br/>getScenes"]
+    E --> K[crypto/jwe-encryptor.ts]
+    E --> L[discovery/dts.ts]
+    E --> M[discovery/bareImports.ts]
+```
+
+### Internal Dependencies
+
+| Package | Role |
+|---------|------|
+| `@auto-engineer/pipeline` | Pipeline server, message bus, command handler registration |
+| `@auto-engineer/narrative` | Scene graph and file discovery for sync (`getScenes`) |
+| `@auto-engineer/file-store` | `NodeFileStore` virtual filesystem abstraction |
+
+### Key Third-party Dependencies
 
-| Endpoint | Description |
-|----------|-------------|
-| `/health` | Server health check |
-| `/registry` | List registered handlers |
-| `/pipeline` | Pipeline state |
-| `/pipeline/diagram` | Visual diagram |
-| `/events` | SSE stream |
-| `ws://<syncPort>` | File sync WebSocket |
-
-### Dependencies
-
-| Package | Usage |
-|---------|-------|
-| `@auto-engineer/pipeline` | Pipeline server infrastructure |
-| `@auto-engineer/narrative` | File discovery for sync |
-| `@auto-engineer/file-store` | Virtual file system |
+| Package | Role |
+|---------|------|
 | `commander` | CLI argument parsing |
-| `socket.io` | WebSocket communication |
-| `chokidar` | File system watching |
+| `socket.io` | WebSocket-based file sync transport |
+| `chokidar` | File system watcher |
+| `ink` / `@inkjs/ui` | Terminal UI rendering (React-based) |
+| `@ngrok/ngrok` | Public tunnel for remote access |
+| `jose` | JWE encryption of AI provider credentials |
+| `jiti` | Runtime TypeScript config loading |
+| `get-port` | Finds available port |
+| `ora` | Terminal spinner |

package/package.json

@@ -44,9 +44,9 @@
     "ora": "^8.0.1",
     "react": "^18.3.1",
     "socket.io": "^4.7.5",
-    "@auto-engineer/file-store": "1.148.0",
-    "@auto-engineer/narrative": "1.148.0",
-    "@auto-engineer/pipeline": "1.148.0"
+    "@auto-engineer/file-store": "1.150.0",
+    "@auto-engineer/narrative": "1.150.0",
+    "@auto-engineer/pipeline": "1.150.0"
   },
   "devDependencies": {
     "@types/node": "^22.0.0",
@@ -54,7 +54,7 @@
     "ink-testing-library": "^4.0.0",
     "socket.io-client": "^4.8.3"
   },
-  "version": "1.148.0",
+  "version": "1.150.0",
   "scripts": {
     "build": "tsc && tsx ../../scripts/fix-esm-imports.ts && chmod +x dist/bin/auto.js",
     "test": "vitest run --reporter=dot",