@counterfact-labs/cli 0.1.3 → 0.2.4

package/README.md

@@ -1,66 +1,200 @@
-# Counterfact command line
+# counterfact-local
 
-Check an agent's prompt, skills and tool definitions from your terminal or coding agent. Requires Node.js 20 or later.
+Counterfact analysis that runs on the customer's machine. In a source checkout, the development runner reads a local
+folder, sends selected analysis inputs directly to Anthropic with your developer key, and saves the results locally.
+The customer CLI archive contains compiled analysis code and a signed encrypted instruction bundle. Relay execution
+needs no provider key or readable instructions locally. The installed command connects through the production gatekeeper and saves only cleaned findings and their provenance
+to the console. An analysis charges the balance directly; it has no price confirmation step. Reviewed public release pins are required before it can run.
 
-```sh
-npm install -g @counterfact-labs/cli
-counterfact-labs login
-counterfact-labs check ./support-agent
+The local and hosted products remain supported. Shared rules and request instructions live in this repository's
+private workspace package, `@counterfact-labs/analysis-core`. A future pinned release will let the hosted product
+use this owner after benchmark parity and a tested rollback; this workspace move does not switch hosted analysis.
+
+## Run a local analysis
+
+Use Node.js 22 or later and install the pinned dependencies:
+
+```bash
+npm ci
 ```
 
-The command lists the selected files and shows the server's price before asking to continue. Use `--yes` after agreeing to that price. Use `--dry-run` to see the files and quote without starting analysis or charging your balance. A quote requires uploading the selected files, including in a dry run.
+Set `ANTHROPIC_API_KEY` in your shell to your Anthropic development key, then run these commands from this checkout:
+
+```bash
+npx tsx cli/check.ts --help
+npx tsx cli/check.ts /path/to/agent
+npx tsx cli/check.ts /path/to/agent --output /path/to/results
+```
+
+Analysis makes paid model calls and needs a provider connection. The command reads source files without modifying
+them, prints a readable summary, and writes `findings.json` and `summary.md` to the agent's `.counterfact/` directory
+by default. `--output` chooses another directory. Reports can contain source excerpts; existing reports at those
+paths are replaced after a completed analysis. Exit status is zero when analysis completes, including when it finds
+problems, and nonzero for invalid arguments, a missing key or a failed run.
+
+For a free run of the complete findings flow using committed recordings, with no key or network access:
 
-```sh
-counterfact-labs check ./support-agent --dry-run
-counterfact-labs check ./support-agent --yes
-counterfact-labs finding <finding-id>
-counterfact-labs balance
-counterfact-labs logout
+```bash
+npm run bench:replay
 ```
 
-Checks return exit code 0 on success, 1 on error and 2 when the balance is too low. The server chooses the files, sets the price and writes the result summary. The first eligible check is free. Failed checks are not charged.
+The replay prints precision, recall and comparison with the saved baseline, and fails if either score leaves its noise
+band. It cannot buy missing model answers. See [how local analysis works](docs/how-local-analysis-works.md) for file
+selection, limits, switches and request construction.
 
-By default, results appear as an analysis in a project in your workspace. A matching GitHub repository uses an existing project. Otherwise the check reuses a project from an earlier check of the same folder or creates one. The results link opens the project's findings for that analysis. New and fixed problems are compared with the agents' open findings there, including earlier console results.
+The current code lives in `packages/core/` (shared rules, types, requests and trusted instructions),
+`cli/` (runner and summary), `core/` (local model-driving steps and orchestration), `adapters/` (runtime I/O), `bundle/` (local
+instruction loading, execution wiring and sealing), and `bench/` (offline parity checks). The
+[canonical analysis guide](packages/core/docs/how-we-analyze-an-agent.md) explains shared rules. Working rules are in
+[AGENTS.md](AGENTS.md); copied-code sources are in [provenance.md](docs/provenance.md).
 
-The tool sends the branch, full commit and whether the checked folder has local edits when Git can report them. Analysis history shows these details with who ran the check. The analysis reads the uploaded files, so its findings show your local copy. Older tool versions that omit branch, commit or local edits still work.
+## Running the tests
 
-The console is at `https://app.counterfactlabs.com`. Sign in again after moving from the old address; saved sign-in belongs to the server address.
+```bash
+npm run typecheck
+npm run lint
+npm test
+```
 
-Add the local MCP server to Codex:
+Tests use scripted answers, fake transports and committed recordings, with no real model calls. The instruction guard
+runs with `npm test`, or by itself:
 
-```sh
-codex mcp add counterfact -- npx -y @counterfact-labs/cli@latest mcp
+```bash
+npx tsx --test core/instruction-text.test.ts core/test/instruction-text.test.ts
 ```
 
-Checks can take several minutes. Codex has a [60-second tool timeout by default](https://learn.chatgpt.com/docs/extend/mcp?surface=cli). In `~/.codex/config.toml`, add `tool_timeout_sec = 900` to the existing `[mcp_servers.counterfact]` section. It should look like this; keep any other settings already in that section:
+## Verify live API acceptance
+
+With `ANTHROPIC_API_KEY` set, this developer command sends one request for every registered template and answer
+format, using the committed fictional Golu example and production model choices:
 
-```toml
-[mcp_servers.counterfact]
-command = "npx"
-args = ["-y", "@counterfact-labs/cli@latest", "mcp"]
-tool_timeout_sec = 900
+```bash
+npx tsx scripts/relay/live-acceptance.ts --budget-ledger /tmp/relay-acceptance-budget.json
 ```
 
-Cancellation stops the local wait. A confirmed check keeps running on the server after cancellation or a timeout. Use the project link from the preview to follow it in the console.
+It prints JSON with each route's acceptance or exact API error and measured dollar cost, and exits nonzero if any
+request was not accepted. Output is limited to 64 tokens per request; incomplete answers still establish API
+acceptance. The persistent ledger reserves counted input and maximum output before spending, with a hard $3 cap.
+Reuse the ledger to preserve that cap across invocations; each run gives its entries a unique run identifier.
+Duplicate entries report `duplicate_entry` rather than `budget`; uncertain charges still stop further spending.
+The ledger holds a `<ledger-file>.lock` file while running. If a run is killed and later runs print
+`Live acceptance failed.`, remove only that lock file after confirming no capture process is still running.
+Keep the ledger itself so earlier spending and uncertain charges remain reserved.
+Second opinions are disabled in local analysis, so that registered route uses the existing Anthropic parity
+fixture's checker model. This command is separate from offline tests and must be run explicitly.
+
+## Use the published CLI
+
+The package is `@counterfact-labs/cli` 0.2.0 and requires Node.js 22 or later. Its executable is
+`counterfact-labs`. Login keeps browser sign-in and adds `--email` and verify-only `--code`. MCP adds
+`sign_in_with_code` and optional email arguments; logout, balance and finding lookup retain their commands and tool names.
+With `CF_EMAIL_SIGN_IN=0`, login help, arguments and the full MCP tool/status contract retain their previous behavior.
+`mcp` exposes `analyze_agent` and reports `analyzeToolAllowed`; the old analysis command and tool names are refused.
+`analyze [path]` runs locally through the relay; `--yes`, `--dry-run`, and the MCP
+`confirm` argument are removed. Each requested analysis runs once without a price prompt. Prices belong on the website
+and top-up page; an uncovered balance refuses with a top-up link.
+
+The archive also ships `bundle/client-release.json` with its public Ed25519 verification key and local/relay versions,
+and the gatekeeper's public production trust policy. Production trust has no environment override. With exactly
+`CF_RELAY_DEV=1`, development startup accepts separate issuer trust, a bundle and public release pins through
+`CF_RELAY_DEV_TRUST`, `CF_RELAY_DEV_BUNDLE` and `CF_RELAY_DEV_RELEASE`. This exception permits only HTTPS loopback
+connections (`localhost`, `127.x.x.x` or `::1`) and does not establish sealed execution.
+The checked-in release pins and production image trust remain unconfigured; installing reviewed pins and ciphertext
+is a release prerequisite.
+
+The npx commands below work once 0.2.0 is tagged `latest`.
+Use npx to run the newest release:
+
+```bash
+npx -y @counterfact-labs/cli@latest login
+npx -y @counterfact-labs/cli@latest balance
+npx -y @counterfact-labs/cli@latest analyze ./agent
+npx -y @counterfact-labs/cli@latest finding <id>
+npx -y @counterfact-labs/cli@latest mcp
+```
 
-Add this to your project's `AGENTS.md`:
+To sign in with an emailed code, run:
 
-```markdown
-First call sign_in_status; if not signed in, call sign_in and then sign_in_status with wait: true.
-Always show the returned sign-in link and code, so I can sign in on any device.
-When I ask to check an agent, use Counterfact's check_agent tool with its folder path.
-Call it without confirm first. Show me the selected files and price, and get my
-agreement before calling again with confirm: true. If the files or price change,
-show me a fresh preview and ask again. Use get_finding for an explanation, quoted
-lines and suggested fixes. Use get_balance for the balance and top-up link.
-After editing prompts, skills or tool definitions, suggest a check and wait for
-me to ask for it.
+```bash
+npx -y @counterfact-labs/cli@latest login --email reader@example.test
 ```
 
-The MCP server reads local files and communicates over stdin and stdout. Its tools are `sign_in_status(wait?)`, `sign_in()`, `check_agent(path, confirm?)`, `get_finding(id)` and `get_balance()`. Sign-in status also reports whether Claude Code settings allow the check tool. Sign-in opens a browser when a local desktop is available and always returns a link and code for signing in on any device. Each result's structured content repeats the tool's text, and completed checks add their finding ids. Keep sign-in and its wait, and the preview and confirmation, in the same MCP session.
+Enter the code from your email when prompted. Without a terminal, this command sends the code and prints how to finish. Complete sign-in with:
+
+```bash
+npx -y @counterfact-labs/cli@latest login --email reader@example.test --code <code>
+```
+
+The command saves your sign-in on this computer. Run login without --email to use browser sign-in.
+
+Passing --code puts the code in process arguments and may leave it in your shell history. Use the hidden terminal prompt to avoid that.
 
-Privacy: only files requested by the server are uploaded. The tool skips `.git`, `node_modules`, `.gitignore` matches, symlinks and secret file names such as `.env` and private keys. Uploaded files are saved with the analysis so findings can show the source that was checked. Sign-in is saved in `~/.config/counterfact-labs/credentials` with owner-only permissions. `logout` removes that local sign-in; revoke the token from your account to disable it elsewhere.
+Set CF_EMAIL_SIGN_IN=0 before starting the command or MCP server to restore the original browser sign-in tools and terminal commands. Restart the MCP server after changing this setting.
 
-Set `COUNTERFACT_URL` to use another server. It defaults to `https://app.counterfactlabs.com`. Sign-in is tied to the server address; sign in again when changing it.
+MCP sign_in_with_code receives the code as a tool argument, so it remains in your MCP client's transcript. Counterfact does not repeat it in tool results, logs, or reports. Use the hidden terminal prompt if you want to keep the code out of the MCP transcript.
+
+Or install globally: `npm install -g @counterfact-labs/cli@latest`.
+
+If you use Claude Code's sandbox, allow relay.counterfactlabs.com and app.counterfactlabs.com when Claude Code asks,
+or add both to the sandbox's allowed domains.
+
+The command uses the existing saved sign-in and sends `X-Counterfact-CLI-Version: 0.2.0`. It asks
+`/api/cli/checks/mode` before reading source or starting paid work. Upload, paused, update-required and unavailable
+responses stop the analysis; the command never falls back to uploading files. Settings are captured at command start.
+The analysis uses a committed Git folder. Admitted files must match that commit; unrelated local output files are ignored.
+The console owns prices, expressed in millionths of a dollar.
+
+The installed command uses the relay address in `bundle/client-release.json` and saved sign-in. The relay runner sends
+repository facts and the admitted file list (paths and sizes, without contents or absolute checkout paths) to
+`POST /api/cli/checks` to start local analysis. The console decides the project and returns its id. Development can
+override `CF_RELAY_URL`; normal use needs no transport switches or provider key. `COUNTERFACT_URL` retains the
+development server override.
+
+Release stays blocked while `relayUrl` and `projectContext` are unconfigured in the checked-in metadata. Set
+`projectContext` to `checks` only after local start support is reviewed. The release validator requires both settings
+along with the signed bundle and trust pins; an environment override cannot bypass the project-context pin.
+
+## Build and release the customer CLI archive
+
+Generate `bundle/sealed.json` with the existing release tooling and externally supplied release keys. Install the
+reviewed public verification key, matching local/relay versions, HTTPS relay origin and reviewed project-context support
+in `bundle/client-release.json`, and the reviewed
+production trust policy in `gatekeeper/trust/production.json`. These trust files cannot be overridden through the
+environment. Then run:
+
+```bash
+npm run release:cli
+```
+
+The release command builds, packs, verifies the archive contents and prints its SHA-256 hash. Only compiled runtime
+modules, the encrypted bundle and release metadata ship, together with npm's mandatory package metadata and README.
+The trusted core remains a development dependency; its readable instructions are excluded from the installed CLI.
+The installed runtime dependencies also exclude TypeScript. The trusted shared-core archive used by hosted
+integrations continues to ship its readable instructions.
+The archive exposes `@counterfact-labs/cli/gatekeeper` for injected verified connections.
+
+The release workflow runs only through a manual GitHub Actions trigger and publishes the verified archive with
+`--tag next`, using the `NPM_TOKEN` secret. It refuses unconfigured bundle and trust pins. No release or publish is
+performed by this implementation task. After the launch decision, move the public default explicitly:
+
+```bash
+npm dist-tag add @counterfact-labs/cli@0.2.0 latest
+```
 
-From this repository, `npm run cli:test` copies the API types and secret-file rule from their server owners, compiles the package with TypeScript and runs its offline tests. No additional npm dependencies are needed.
+The source-only `cli/check.ts` development runner accepts `CF_LOCAL_RELAY=1`, `CF_RELAY_URL`, `CF_CONSOLE_URL`,
+and `CF_CONSOLE_TOKEN`. Other switch values retain direct source execution. The installed command
+uses the saved sign-in and release configuration described above. Completed runs save local reports and a private
+`completion.json` record containing the exact findings payload and analysis identity, with no bearer or reservation
+capability. A pending upload is retried on the next invocation using current credentials. The source runner's
+`--retry-upload` explicitly retries a previously completed upload without buying another analysis. Keep this record
+private because findings can contain excerpts. The source runner prints the saved analysis id and project findings
+link; the installed command prints finding ids and the full-results link. Failures report only trusted settlement/status
+facts; a local error cannot promise a refund. Customer source
+files travel only through the gatekeeper to the attested relay. Relay failures stop the run.
+
+Offline package tests install real archives in a clean temporary directory, inspect the installed dependencies for
+readable instructions, run the linked executable and complete the analysis stages with a scripted provider and an
+injected verified connection. A separate end-to-end test uses real loopback gatekeeper TLS and the real relay runtime.
+Server storage and settlement still require the disposable PostgreSQL integration run.
+Reviewed server compatibility for variable-size check batches, split scope limits, settings-selected variants and the
+optional headline stage remains a launch gate; offline fakes do not establish those production contracts.

package/bundle/client-release.json

@@ -0,0 +1,7 @@
+{
+  "verificationKey": "-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VwAyEAbBab1PYdmBKXxCvoJ47toJhQvxPevo9nlZnb0zbDWoc=\n-----END PUBLIC KEY-----\n",
+  "relayVersion": "relay-1.0.1",
+  "localVersion": "0.2.4",
+  "relayUrl": "https://relay.counterfactlabs.com:8443",
+  "projectContext": "checks"
+}