@skillrecordings/cli 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,35 @@
+# @skillrecordings/cli
+
+## 0.1.0
+
+### Minor Changes
+
+- 5734b19: Make CLI public for npm publishing
+
+## 0.0.4
+
+### Patch Changes
+
+- Updated dependencies [7ae3e99]
+  - @skillrecordings/sdk@0.6.0
+
+## 0.0.3
+
+### Patch Changes
+
+- Updated dependencies [36efccf]
+  - @skillrecordings/sdk@0.5.0
+
+## 0.0.2
+
+### Patch Changes
+
+- Updated dependencies [7c5c5d8]
+  - @skillrecordings/sdk@0.4.0
+
+## 0.0.1
+
+### Patch Changes
+
+- Updated dependencies [2820cb9]
+  - @skillrecordings/core@0.0.1

package/README.md

@@ -0,0 +1,214 @@
+# @skillrecordings/cli
+
+CLI for the support platform. Agent-friendly with non-interactive defaults.
+
+## Usage
+
+```bash
+bunx @skillrecordings/cli <command> [options]
+
+# or with direct import
+skill <command> [options]
+```
+
+All commands support `--json` for machine-readable output and reliable exit
+codes.
+
+## Commands
+
+### `skill init <name>`
+
+Initialize a new app integration with webhook secret.
+
+```bash
+# Interactive (terminal only)
+skill init
+
+# Non-interactive (required for agents/scripts)
+skill init my-app
+
+# JSON output
+skill init my-app --json
+```
+
+**Options:**
+- `--json` - Output result as JSON (machine-readable)
+
+**Exit codes:**
+- `0` - Success
+- `1` - Error (name required in non-interactive mode, etc.)
+
+### `skill health <slug|url>`
+
+Test integration endpoint health.
+
+```bash
+# Using database lookup (recommended)
+skill health total-typescript
+
+# Direct URL mode
+skill health https://example.com --secret whsec_xxx
+
+# List registered apps
+skill health --list
+
+# JSON output (for agents)
+skill health total-typescript --json
+```
+
+**Options:**
+- `-s, --secret <secret>` - Webhook secret (required for direct URL mode)
+- `-l, --list` - List all registered apps
+- `--json` - Output result as JSON (machine-readable)
+
+**Exit codes:**
+- `0` - Health check passed
+- `1` - Health check failed or error
+
+**JSON output structure:**
+```json
+{
+  "success": true,
+  "endpoint": "https://...",
+  "status": "ok",
+  "responseTime": 730,
+  "actions": [
+    { "name": "lookupUser", "status": "ok" },
+    { "name": "getPurchases", "status": "ok" }
+  ],
+  "summary": { "ok": 4, "notImplemented": 1, "errors": 0 }
+}
+```
+
+### `skill eval <type> <dataset>`
+
+Run evals against a dataset (e.g., routing classifier, canned response
+matcher).
+
+```bash
+# Run routing eval with defaults
+skill eval routing path/to/dataset.json
+
+# With strict thresholds
+skill eval routing dataset.json --min-precision 0.95 --min-recall 0.97
+
+# JSON output for automation
+skill eval routing dataset.json --json
+
+# Custom thresholds
+skill eval routing dataset.json \
+  --min-precision 0.92 \
+  --min-recall 0.95 \
+  --max-fp-rate 0.03 \
+  --max-fn-rate 0.02
+```
+
+**Arguments:**
+- `type` - Eval type (e.g., `routing`)
+- `dataset` - Path to JSON dataset file
+
+**Options:**
+- `--json` - Output result as JSON (machine-readable)
+- `--min-precision <number>` - Minimum precision threshold (default: 0.92)
+- `--min-recall <number>` - Minimum recall threshold (default: 0.95)
+- `--max-fp-rate <number>` - Maximum false positive rate (default: 0.03)
+- `--max-fn-rate <number>` - Maximum false negative rate (default: 0.02)
+
+**Exit codes:**
+- `0` - All metrics passed thresholds
+- `1` - One or more metrics below threshold or error
+
+**Output includes:**
+- Precision, recall, false positive/negative rates
+- Latency percentiles (p50, p95, p99)
+- Token usage and estimated cost
+- Category-level breakdown (if applicable)
+
+## App Onboarding Workflow
+
+Typical flow for adding a new app integration:
+
+```bash
+# 1. Initialize with app name
+skill init my-app --json
+# Returns: { "success": true, "appName": "my-app",
+#            "webhookSecret": "whsec_xxx" }
+
+# 2. Register webhook endpoint in your app
+# Save the webhook secret and configure your endpoint to:
+# POST /api/support-webhooks with Authorization: Bearer whsec_xxx
+
+# 3. Test health before going live
+skill health my-app
+# Verifies: endpoint reachable, signature verification works,
+# actions implemented
+
+# 4. Run evals (optional, for routing/matching logic)
+skill eval routing path/to/labeled-dataset.json --json
+
+# 5. Deploy and monitor
+# Check logs via Axiom/Langfuse for inbound messages
+```
+
+All commands work non-interactively and report errors with exit codes
+(0=success, 1=error).
+
+## Agent Usage
+
+All commands support `--json` for machine-readable output and
+non-interactive operation:
+
+**init command:**
+- Requires `name` argument (non-interactive mode)
+- Returns JSON: `{ "success": true, "appName": "...",
+  "webhookSecret": "whsec_..." }`
+- Use `--json` for reliable parsing
+
+**health command:**
+- Use `--json` for JSON output (structured for parsing)
+- Use `--list` to discover all registered apps
+- Returns exit code 0 if healthy, 1 if any check fails
+
+**eval command:**
+- Requires `type` and `dataset` arguments
+- Accepts custom threshold gates (precision, recall, false
+  positive/negative rates)
+- Returns exit code 0 if all metrics pass, 1 otherwise
+- Use `--json` for machine-readable report
+
+**Error handling:**
+- All commands output `{ "success": false, "error": "message" }` on
+  JSON mode
+- Check exit codes: 0 = success, 1 = error
+- Never interactive in non-TTY environments (CI/CD safe)
+
+## Auth (Encrypted Secrets)
+
+Distribute CLI secrets to team members via age encryption + 1Password.
+
+```bash
+# Generate keypair (admin)
+skill auth keygen
+
+# Encrypt secrets (admin)
+skill auth encrypt .env.local
+
+# Decrypt secrets (team)
+skill auth decrypt .env.local.age --output .env.local
+```
+
+See [docs/CLI-AUTH.md](./docs/CLI-AUTH.md) for complete setup guide.
+
+## Implementation
+
+- `packages/cli/src/commands/` - Command implementations
+- `packages/cli/src/index.ts` - CLI entry point
+- Entry point: `#!/usr/bin/env bun` (runs with Bun directly)
+
+## Do / Don't
+
+- Do use `--json` flag for automation/agents/scripts
+- Do check exit codes in shell scripts
+- Do pass `name` argument to `init` in CI/CD (non-interactive required)
+- Don't rely on interactive prompts outside terminal
+- Don't parse stdout (use `--json` for structured output)

package/bin/skill.ts

@@ -0,0 +1,3 @@
+#!/usr/bin/env bun
+import '../preload'
+import '../src/index'

package/data/tt-archive-dataset.json

@@ -0,0 +1 @@
+[]

package/data/validate-test-dataset.json

@@ -0,0 +1,97 @@
+[
+  {
+    "id": "validate_clean_response",
+    "name": "Clean response - should pass validation",
+    "draft": "Here's a login link for your account: https://example.com/login?token=abc123\n\nYou should have full access to the course now.",
+    "assertions": {
+      "noFabrication": true,
+      "noMetaCommentary": true,
+      "noInternalLeak": true,
+      "noBannedPhrases": true
+    },
+    "context": {
+      "customer": "recent-purchase"
+    }
+  },
+  {
+    "id": "validate_internal_leak",
+    "name": "Internal leak - exposes routing info",
+    "draft": "No instructor routing configured for this app. You'll want to reach out through another channel.",
+    "assertions": {
+      "noInternalLeak": true
+    },
+    "context": {
+      "customer": "no-purchase"
+    }
+  },
+  {
+    "id": "validate_meta_commentary",
+    "name": "Meta commentary - agent explains itself",
+    "draft": "This is clearly a vendor outreach email, not a support request. I won't draft a response here since it falls outside customer support scope.",
+    "assertions": {
+      "noMetaCommentary": true
+    },
+    "context": {
+      "customer": "no-purchase"
+    }
+  },
+  {
+    "id": "validate_banned_phrases",
+    "name": "Banned phrases - corporate speak",
+    "draft": "Great! I'd recommend starting with the basics. Let me know if you have any other questions! I hope this helps!",
+    "assertions": {
+      "noBannedPhrases": true
+    },
+    "context": {
+      "customer": "recent-purchase"
+    }
+  },
+  {
+    "id": "validate_fabrication",
+    "name": "Fabrication - made up course content",
+    "draft": "Start with the fundamentals section in Module 1. It covers core concepts like typing basics, generics, and utility types.",
+    "assertions": {
+      "noFabrication": true,
+      "mustNotContain": ["fundamentals section", "Module 1"]
+    },
+    "context": {
+      "customer": "no-purchase"
+    }
+  },
+  {
+    "id": "validate_em_dash",
+    "name": "Em dash - banned character",
+    "draft": "TypeScript — like JavaScript — is a powerful language for building web applications.",
+    "assertions": {
+      "noBannedPhrases": true
+    },
+    "context": {
+      "customer": "recent-purchase"
+    }
+  },
+  {
+    "id": "validate_multiple_issues",
+    "name": "Multiple issues - several patterns detected",
+    "draft": "Great! I understand your frustration. The routing failed so I can't process this. Start with the fundamentals section — it covers everything.",
+    "assertions": {
+      "noFabrication": true,
+      "noInternalLeak": true,
+      "noBannedPhrases": true
+    },
+    "context": {
+      "customer": "no-purchase"
+    }
+  },
+  {
+    "id": "validate_outside_scope",
+    "name": "Outside scope leak",
+    "draft": "This falls outside my scope as a support agent. You should contact the instructor directly.",
+    "assertions": {
+      "noInternalLeak": true,
+      "mustNotContain": ["outside", "scope"]
+    },
+    "context": {
+      "customer": "no-purchase"
+    }
+  }
+]