@skillrecordings/cli 0.19.0 → 0.20.0

package/README.md

@@ -1,385 +1,171 @@
 # @skillrecordings/cli
 
-CLI for the support platform. Agent-friendly with non-interactive defaults.
+Agent-friendly CLI for the Skill Recordings support platform.
 
-## Install
+## Setup (2 minutes)
+
+You need: a GitHub account in the `skillrecordings` org.
 
 ```bash
+# 1. Install
 curl -fsSL https://raw.githubusercontent.com/skillrecordings/support/main/packages/cli/install.sh | bash
-```
 
-Installs to `~/.local/bin/skill`. Add to your PATH if needed:
-
-```bash
+# 2. Add to PATH (if not already)
 echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc && source ~/.zshrc
-```
-
-After install, run `skill auth setup` to configure credentials.
-
-## Usage
-
-```bash
-bunx @skillrecordings/cli <command> [options]
-
-# or with direct import
-skill <command> [options]
-```
-
-All commands support `--json` for machine-readable output and reliable exit
-codes.
 
-## 1Password Quickstart (60 seconds)
-
-If you already have access to the Support vault:
-
-```bash
-# 1) Install required CLI
-brew install 1password-cli
-
-# 2) Optional but recommended: cache OP token in local secrets CLI
-secrets add skill_support_1password_service_account_token
-
-# 3) Run one setup command
+# 3. Authenticate (opens browser, no other tools needed)
 skill auth setup
 
-# 4) Verify
+# 4. Verify everything works
 skill doctor
 ```
 
-Notes:
-- `secrets` CLI is optional. If missing, `skill auth setup` falls back to `op signin`.
-- Setup writes `SKILL_AGE_KEY` to `~/.config/skill/age.key` so `.env.encrypted` works without keychain.
-
-## OAuth Broker Spike
-
-Use this to inspect the planned GitHub OAuth + broker auth model that reduces
-local dependencies for team members:
-
-```bash
-skill auth oauth-spike
-skill auth oauth-spike --json
-```
-
-## Adaptive Hints
-
-The CLI prints adaptive onboarding/discovery hints to `stderr` for new users.
-Hints learn from usage and fade as you run more commands.
+That's it. No 1Password CLI, no age keys, no manual secret management.
 
-**Opt out:**
-- Use `--quiet`
-- Use `--json`
-- Pipe output (non-TTY)
+The CLI authenticates via GitHub device flow → the broker verifies your org
+membership → secrets are delivered encrypted and held in memory only.
 
 ## Commands
 
-### `skill init <name>`
-
-Initialize a new app integration with webhook secret.
-
 ```bash
-# Interactive (terminal only)
-skill init
-
-# Non-interactive (required for agents/scripts)
-skill init my-app
-
-# JSON output
-skill init my-app --json
+skill <command> [options]
 ```
 
-**Options:**
-- `--json` - Output result as JSON (machine-readable)
+All commands support `--json` for machine-readable output.
 
-**Exit codes:**
-- `0` - Success
-- `1` - Error (name required in non-interactive mode, etc.)
+| Command | What it does |
+|---|---|
+| `skill auth setup` | One-command bootstrap (GitHub device flow) |
+| `skill auth status` | Session + provider status |
+| `skill doctor` | Deep health check with remediation hints |
+| `skill health <app>` | Test app webhook endpoint |
+| `skill health --list` | List registered apps |
+| `skill init <name>` | Initialize new app integration |
+| `skill eval routing <dataset>` | Run routing classifier evals |
+| `skill auth oauth-spike` | Inspect broker readiness |
 
-### `skill health <slug|url>`
+## How Auth Works
 
-Test integration endpoint health.
-
-```bash
-# Using database lookup (recommended)
-skill health total-typescript
-
-# Direct URL mode
-skill health https://example.com --secret whsec_xxx
-
-# List registered apps
-skill health --list
-
-# JSON output (for agents)
-skill health total-typescript --json
-```
-
-**Options:**
-- `-s, --secret <secret>` - Webhook secret (required for direct URL mode)
-- `-l, --list` - List all registered apps
-- `--json` - Output result as JSON (machine-readable)
-
-**Exit codes:**
-- `0` - Health check passed
-- `1` - Health check failed or error
-
-**JSON output structure:**
-```json
-{
-  "success": true,
-  "endpoint": "https://...",
-  "status": "ok",
-  "responseTime": 730,
-  "actions": [
-    { "name": "lookupUser", "status": "ok" },
-    { "name": "getPurchases", "status": "ok" }
-  ],
-  "summary": { "ok": 4, "notImplemented": 1, "errors": 0 }
-}
 ```
-
-### `skill eval <type> <dataset>`
-
-Run evals against a dataset (e.g., routing classifier, canned response
-matcher).
-
-```bash
-# Run routing eval with defaults
-skill eval routing path/to/dataset.json
-
-# With strict thresholds
-skill eval routing dataset.json --min-precision 0.95 --min-recall 0.97
-
-# JSON output for automation
-skill eval routing dataset.json --json
-
-# Custom thresholds
-skill eval routing dataset.json \
-  --min-precision 0.92 \
-  --min-recall 0.95 \
-  --max-fp-rate 0.03 \
-  --max-fn-rate 0.02
+skill auth setup
+    → GitHub device flow (approve in browser)
+    → Broker verifies skillrecordings org membership
+    → Short-lived session tokens issued (15min access / 8hr refresh)
+    → skill auth exec decrypts secrets in memory per-command
 ```
 
-**Arguments:**
-- `type` - Eval type (e.g., `routing`)
-- `dataset` - Path to JSON dataset file
-
-**Options:**
-- `--json` - Output result as JSON (machine-readable)
-- `--min-precision <number>` - Minimum precision threshold (default: 0.92)
-- `--min-recall <number>` - Minimum recall threshold (default: 0.95)
-- `--max-fp-rate <number>` - Maximum false positive rate (default: 0.03)
-- `--max-fn-rate <number>` - Maximum false negative rate (default: 0.02)
+**Broker endpoints** (on `skill-support-agent-front.vercel.app`):
 
-**Exit codes:**
-- `0` - All metrics passed thresholds
-- `1` - One or more metrics below threshold or error
+| Endpoint | Purpose |
+|---|---|
+| `POST /api/auth/device/start` | Start GitHub device flow |
+| `POST /api/auth/device/poll` | Exchange device code for session |
+| `POST /api/auth/session/refresh` | Refresh session, re-check membership |
+| `POST /api/env/materialize` | Age-encrypted env delivery |
 
-**Output includes:**
-- Precision, recall, false positive/negative rates
-- Latency percentiles (p50, p95, p99)
-- Token usage and estimated cost
-- Category-level breakdown (if applicable)
+**Legacy path:** 1Password CLI + `skill auth setup --legacy` still works as
+admin/break-glass mode.
 
-## App Onboarding Workflow
-
-Typical flow for adding a new app integration:
+## App Onboarding
 
 ```bash
-# 1. Initialize with app name
+# 1. Initialize
 skill init my-app --json
-# Returns: { "success": true, "appName": "my-app",
-#            "webhookSecret": "whsec_xxx" }
 
-# 2. Register webhook endpoint in your app
-# Save the webhook secret and configure your endpoint to:
+# 2. Register webhook in your app (use returned secret)
 # POST /api/support-webhooks with Authorization: Bearer whsec_xxx
 
-# 3. Test health before going live
+# 3. Verify
 skill health my-app
-# Verifies: endpoint reachable, signature verification works,
-# actions implemented
-
-# 4. Run evals (optional, for routing/matching logic)
-skill eval routing path/to/labeled-dataset.json --json
 
-# 5. Deploy and monitor
-# Check logs via Axiom/Langfuse for inbound messages
+# 4. Run evals (optional)
+skill eval routing labeled-dataset.json --json
 ```
 
-All commands work non-interactively and report errors with exit codes
-(0=success, 1=error).
-
-## Agent Usage
-
-All commands support `--json` for machine-readable output and
-non-interactive operation:
-
-**init command:**
-- Requires `name` argument (non-interactive mode)
-- Returns JSON: `{ "success": true, "appName": "...",
-  "webhookSecret": "whsec_..." }`
-- Use `--json` for reliable parsing
-
-**health command:**
-- Use `--json` for JSON output (structured for parsing)
-- Use `--list` to discover all registered apps
-- Returns exit code 0 if healthy, 1 if any check fails
-
-**eval command:**
-- Requires `type` and `dataset` arguments
-- Accepts custom threshold gates (precision, recall, false
-  positive/negative rates)
-- Returns exit code 0 if all metrics pass, 1 otherwise
-- Use `--json` for machine-readable report
-
-**Error handling:**
-- All commands output `{ "success": false, "error": "message" }` on
-  JSON mode
-- Check exit codes: 0 = success, 1 = error
-- Never interactive in non-TTY environments (CI/CD safe)
-
-## Secrets Management
+## Health Check
 
-The CLI uses a layered secrets system:
-
-1. **1Password (preferred)** - Service account token resolves secrets directly
-2. **Encrypted `.env.encrypted`** - Age-encrypted env file for offline/CI use
-3. **Plain `.env.local`** - Local development fallback
+```bash
+# Quick check
+skill health total-typescript
 
-### Secret Resolution Order
+# All apps
+skill health --list
 
-```
-1Password (OP_SERVICE_ACCOUNT_TOKEN set?)
-    ↓ yes → resolve from 1Password vault
-    ↓ no
-.env.encrypted exists + SKILL_AGE_KEY available?
-    ↓ yes → decrypt and load
-    ↓ no
-.env.local exists?
-    ↓ yes → load plain env vars
-    ↓ no → error: missing secrets
+# Deep system check
+skill doctor --json
 ```
 
-### Adding a New Secret
+## Agent Usage
 
-**Step 1: Add to `secret-refs.ts`**
+- `--json` on every command for structured output
+- `--quiet` suppresses adaptive hints
+- Exit codes: `0` = success, `1` = error
+- Non-interactive in non-TTY environments (CI/CD safe)
+- Error shape: `{ "success": false, "error": "message" }`
 
-```typescript
-// packages/cli/src/core/secret-refs.ts
-export const SECRET_REFS = {
-  // ... existing secrets
-  MY_NEW_KEY: 'op://Support/skill-cli/MY_NEW_KEY',
-} as const
-```
+## Secrets Management
 
-**Step 2: Add to 1Password**
+The broker handles secrets for most users. Admins who need direct access:
 
-```bash
-# Using op CLI
-op item edit "skill-cli" --vault "Support" "MY_NEW_KEY=your-secret-value"
+### Resolution Order
 
-# Or via 1Password UI:
-# 1. Open Support vault → skill-cli item
-# 2. Add new field: MY_NEW_KEY = your-value
 ```
-
-**Step 3: Update `.env.encrypted`**
-
-```bash
-# Decrypt current secrets
-AGE_KEY=$(op read "op://Support/skill-cli-age-key/private_key")
-age -d -i <(echo "$AGE_KEY") .env.encrypted > .env.local
-
-# Add new secret to .env.local
-echo "MY_NEW_KEY=your-secret-value" >> .env.local
-
-# Re-encrypt
-AGE_PUB=$(echo "$AGE_KEY" | age-keygen -y)
-age -r "$AGE_PUB" .env.local > .env.encrypted
-
-# Verify
-age -d -i <(echo "$AGE_KEY") .env.encrypted | grep MY_NEW_KEY
+OAuth Broker (default, v0.19.0+)
+    → GitHub auth → broker decrypts → ephemeral age envelope → memory only
+1Password (admin/break-glass)
+    → OP_SERVICE_ACCOUNT_TOKEN → resolve from vault
+.env.encrypted (CI fallback)
+    → SKILL_AGE_KEY → decrypt and load
+.env.local (local dev fallback)
+    → load plain env vars
 ```
 
-**Step 4: Commit changes**
+### Adding a Secret
 
-```bash
-git add packages/cli/src/core/secret-refs.ts packages/cli/.env.encrypted
-git commit -m "chore(cli): add MY_NEW_KEY secret"
-```
+1. Add ref to `packages/cli/src/core/secret-refs.ts`
+2. Add value to 1Password: `op item edit "skill-cli" --vault "Support" "MY_KEY=value"`
+3. Update `.env.encrypted` (see below)
+4. Commit: `git add secret-refs.ts .env.encrypted`
 
-### Updating an Existing Secret
+### Updating `.env.encrypted`
 
 ```bash
-# 1. Update in 1Password
-op item edit "skill-cli" --vault "Support" "MY_KEY=new-value"
-
-# 2. Update .env.encrypted (same process as adding)
 AGE_KEY=$(op read "op://Support/skill-cli-age-key/private_key")
 age -d -i <(echo "$AGE_KEY") .env.encrypted > .env.local
-
-# Edit .env.local with new value
-sed -i '' 's/MY_KEY=.*/MY_KEY=new-value/' .env.local
-
-# Re-encrypt
+# edit .env.local
 AGE_PUB=$(echo "$AGE_KEY" | age-keygen -y)
 age -r "$AGE_PUB" .env.local > .env.encrypted
-```
-
-### Auth Commands
-
-```bash
-# Check current auth status
-skill auth status
-
-# Validate 1Password token
-skill auth login
-
-# Show service account info
-skill auth whoami
-
-# Interactive setup wizard
-skill auth setup
+rm .env.local
 ```
 
 ### Key Locations
 
 | Item | Location |
-|------|----------|
+|---|---|
 | Secrets | `op://Support/skill-cli/*` |
 | Age keypair | `op://Support/skill-cli-age-key/private_key` |
 | Encrypted env | `packages/cli/.env.encrypted` |
 | Secret refs | `packages/cli/src/core/secret-refs.ts` |
 
-### CI/CD Usage
-
-For CI environments without 1Password:
+### CI/CD
 
 ```bash
-# Set age key as CI secret, then:
+# With 1Password service account
+export OP_SERVICE_ACCOUNT_TOKEN="$OP_TOKEN"
+skill auth status
+
+# Or with age key
 echo "$SKILL_AGE_KEY" > /tmp/age.key
 age -d -i /tmp/age.key .env.encrypted > .env.local
 rm /tmp/age.key
 ```
 
-Or use 1Password service account:
+## Adaptive Hints
 
-```bash
-export OP_SERVICE_ACCOUNT_TOKEN="$OP_TOKEN"
-skill auth status  # Verifies connection
-skill front inbox  # Commands auto-resolve secrets
-```
+New users see contextual onboarding hints on `stderr`. They fade with usage.
+Suppress with `--quiet`, `--json`, or piped output.
 
 ## Implementation
 
-- `packages/cli/src/commands/` - Command implementations
-- `packages/cli/src/index.ts` - CLI entry point
-- Entry point: `#!/usr/bin/env bun` (runs with Bun directly)
-
-## Do / Don't
-
-- Do use `--json` flag for automation/agents/scripts
-- Do check exit codes in shell scripts
-- Do pass `name` argument to `init` in CI/CD (non-interactive required)
-- Don't rely on interactive prompts outside terminal
-- Don't parse stdout (use `--json` for structured output)
+- `packages/cli/src/commands/` — command implementations
+- `packages/cli/src/index.ts` — entry point (`#!/usr/bin/env bun`)