@dutchmanlabs/evalstudio 0.1.0

package/README.md

@@ -0,0 +1,106 @@
+# Eval Studio CLI
+
+Local-first CLI for scanning AI agents, generating eval suites through Dutchman Labs, running them locally, and uploading results back to Eval Studio.
+
+## Install and Run
+
+From the monorepo during development:
+
+```bash
+npm run build:cli
+node packages/cli/dist/index.js --help
+node packages/cli/dist/index.js login
+```
+
+After install, the quick start is:
+
+```bash
+evalstudio login
+evalstudio init
+evalstudio detect
+evalstudio generate
+evalstudio run
+```
+
+Users will be able to get that command UX either via `npx`:
+
+```bash
+npx @dutchmanlabs/evalstudio@latest login
+npx @dutchmanlabs/evalstudio@latest init
+npx @dutchmanlabs/evalstudio@latest detect
+npx @dutchmanlabs/evalstudio@latest generate
+npx @dutchmanlabs/evalstudio@latest run
+```
+
+Or by installing globally:
+
+```bash
+npm install -g @dutchmanlabs/evalstudio
+evalstudio --help
+```
+
+## Commands
+
+- `evalstudio login`
+- `evalstudio init`
+- `evalstudio detect`
+- `evalstudio scan` (alias)
+- `evalstudio generate`
+- `evalstudio run`
+- `evalstudio status`
+- `evalstudio export`
+
+## Local Files
+
+The CLI writes state in the current repo under `.evalstudio/`:
+
+- `.evalstudio/config.json`
+- `.evalstudio/scan-results.json`
+- `.evalstudio/latest-suite.json`
+- `.evalstudio/latest-run.json`
+- `.evalstudio/exports/`
+
+Global auth is stored in `~/.evalstudio/config.json`.
+
+`generate` writes the current hosted suite to `.evalstudio/latest-suite.json`.
+
+`run` executes the suite locally, saves the result set to `.evalstudio/latest-run.json`, and then uploads those results to the Dutchman Labs dashboard.
+
+`export` is local-only. It transforms `.evalstudio/latest-run.json` into JSONL, CSV, or pytest artifacts under `.evalstudio/exports/`.
+
+## Help
+
+```bash
+evalstudio --help
+evalstudio help
+evalstudio generate --help
+evalstudio help run
+```
+
+## Demo Target
+
+The canonical sibling demo repo used during validation is:
+
+`/Users/riyadsarsour/Desktop/dutchman/testagent`
+
+That demo agent listens on:
+
+`http://127.0.0.1:3000/api/chat`
+
+Typical demo flow:
+
+```bash
+cd /Users/riyadsarsour/Desktop/dutchman/testagent
+node /Users/riyadsarsour/Desktop/dutchman/dutchmanlabs/packages/cli/dist/index.js init
+node /Users/riyadsarsour/Desktop/dutchman/dutchmanlabs/packages/cli/dist/index.js detect
+node /Users/riyadsarsour/Desktop/dutchman/dutchmanlabs/packages/cli/dist/index.js generate
+node /Users/riyadsarsour/Desktop/dutchman/dutchmanlabs/packages/cli/dist/index.js run --url http://127.0.0.1:3000/api/chat
+node /Users/riyadsarsour/Desktop/dutchman/dutchmanlabs/packages/cli/dist/index.js export
+```
+
+Artifacts to inspect after the demo:
+
+- `.evalstudio/scan-results.json`
+- `.evalstudio/latest-suite.json`
+- `.evalstudio/latest-run.json`
+- `.evalstudio/exports/`

package/VALIDATION.md

@@ -0,0 +1,69 @@
+# Eval Studio CLI Validation
+
+Validation date: 2026-03-31
+
+## Checklist
+
+- [x] Happy path
+- [x] Invalid API key
+- [x] Generation limit exceeded
+- [x] Multiple candidates
+- [x] Local endpoint down
+- [x] Export after run
+
+## Results
+
+### Happy path
+
+- Environment: release-readiness pass used the local mock backend at `http://127.0.0.1:8791` plus a real local HTTP agent endpoint at `http://127.0.0.1:3000/api/chat`
+- Commands exercised: `init`, `detect --candidate 1`, `generate`, `run --url http://127.0.0.1:3000/api/chat --payload '{"input":"{{prompt}}"}'`, `status`, `export`
+- Result: end-to-end success on the polished build
+- Notes:
+  - project created: `proj_72c7f75e`
+  - detect found 2 candidates and selected `app/api/chat/route.ts`
+  - suite created: `suite_3b2835ae`
+  - run uploaded: `run_08bd0e4c`
+  - JSONL, CSV, and pytest exports written locally under `.evalstudio/exports/`
+
+Additional note:
+  - the core product loop was also validated earlier on 2026-03-31 against the real hosted backend with a real API key and local sibling demo repo
+
+### Invalid API key
+
+- Environment: temp home directory with a bogus `es_live_...` key against the real hosted backend
+- Command exercised: `status`
+- Result: CLI shows a human-readable invalid/revoked key error without a stack trace
+
+### Generation limit exceeded
+
+- Environment: local mock backend returning `429 generation_limit_exceeded`
+- Command exercised: `generate`
+- Result: CLI shows a friendly daily limit message and points the user to `evalstudio status`
+
+### Multiple candidates
+
+- Environment: temp validation repo with two detected candidates
+- Command exercised: `detect --candidate 1`
+- Result: CLI prints a ranked list, supports clean selection by number, and stores the selected candidate in `.evalstudio/config.json`
+
+### Local endpoint down
+
+- Environment: valid local suite with no service listening at the configured URL
+- Command exercised: `run --url http://127.0.0.1:3999/api/chat --payload '{"input":"{{prompt}}"}'`
+- Result: CLI surfaces an actionable local endpoint error instead of a raw fetch failure
+
+### Export after run
+
+- Environment: local run cache present
+- Command exercised: `export`
+- Result: CLI writes JSONL, CSV, and pytest artifacts and prints their saved paths
+
+## Additional error smoke tests
+
+- Missing API key: `status` reports `No Eval Studio API key is saved on this machine.`
+- Project not initialized: `status` reports `This repo is not initialized for Eval Studio yet.`
+- No candidates found: `detect` on an empty repo reports `No likely AI agent candidates were found in this repo.`
+- No candidate selected: `generate` on an initialized repo without detection results reports `No candidate selected.`
+- No eval suite generated: `run` on an initialized repo without a suite reports `No eval suite is saved for this repo.`
+- Malformed local agent response: `run` against a server returning `{"ok":true}` reports `Your local agent responded, but the response shape wasn't recognized.`
+- Backend upload failure: `run` against a backend stub that fails result uploads reports `Eval Studio couldn't upload your run results.` and leaves `.evalstudio/latest-run.json` with `"uploadStatus": "pending"`