@haystackeditor/cli 0.17.0 → 0.18.0

package/README.md

@@ -80,41 +80,83 @@ haystack triage <ref> --json    # poll findings later / after --no-wait
 `dismiss`, `mark_reviewed`, `undismiss`, `request_review`, `trigger_review`,
 and `schema`.
 
-**`haystack verify`** verifies the current change using the authenticated
-Haystack fleet. Inside a git checkout, push your commit and run:
+**`haystack verify`** shows the product blast radius of your current change:
+where it shows up in the running app, and what broke. Inside a git checkout
+(nothing needs to be committed or pushed), run:
 
 ```bash
 haystack verify
+haystack verify --no-wait
 haystack verify --json
 ```
 
-The repository defaults to `owner/repo` from the `origin` remote URL. The head
-is the current `HEAD` commit. The base is `git merge-base HEAD
-origin/<default branch>`, using `origin/HEAD` to discover the default branch
-and falling back to `origin/main`. If the current commit is not reachable from
-any remote-tracking branch, the command asks you to push first.
-Explicit `--repo`, `--base`, and `--head` values override their defaults; supplying
-all three also works outside a checkout:
-
-```bash
-haystack verify --repo owner/repo --base main --head feature --json
-```
-
-Intent is optional: `--intent-file <path>` accepts JSON with exactly `problem`,
-`goal`, and `intended_outcomes`. The CLI uses your saved account and waits up to
-35 minutes by default. Pass `--no-wait` to return after the fleet run is queued.
-Repeating the same repository, refs, and intent reuses the same run; choose a new
-`--idempotency-key` when a fresh execution is intentional. A wait that expires
-prints `timed_out: true` and `next_command`, then exits 2. Resume with:
+It captures the checkout, committed and uncommitted changes together, exactly
+as the stop hook does: the repository from `origin`, the base at `git
+merge-base HEAD origin/<default branch>`, and the working tree as a
+deterministic snapshot commit, so the same tree always names the same capture.
+It then finds your crawl of that exact capture: the same code and the same
+change title (HEAD's subject, which tells the crawl's judge what the change is;
+committing or rewording keeps the code but changes the title). When there is
+none, or the one it finds was cancelled, stopped before finishing or is being
+cancelled, it submits the capture through the same request `haystack verify
+precompute` sends (the service starts the crawl, or runs the stopped one again
+once it has shut down), says so in one line, and follows the crawl the service
+acknowledged; it never shows a crawl of another revision or another title.
+
+A crawl builds the app with and without your change, starts from the changed
+code, reaches it in the running app, explores outward with both builds side by
+side, and double-checks and judges every difference. `haystack verify` waits
+until the crawl finishes, printing each step as it happens; there is no time
+limit, and Ctrl-C stops waiting, never the crawl. It then prints:
+
+- a headline: how many bugs it found, none, or why the crawl could not finish;
+- **Where your change shows up in the app**: every changed spot as
+  `file:line`, how far the crawl got with it (never reached, loaded but never
+  ran, shown on screen, ran, ran when its control was pressed, a difference
+  seen, a difference seen and confirmed, judged a bug), and the steps and page
+  that reached it;
+- **What the crawl found**: bugs first, each with a one-sentence reason and the
+  steps to see it;
+- **What the crawl never ran**: changed files no path ran, and how many of the
+  functions your change affects ran.
+
+`--no-wait` prints the crawl's current state and returns. `--interval
+<seconds>` sets the polling interval (default 5), `--account <login>` picks a
+saved account, and `--repo owner/repo` names the repository when `origin` does
+not. `--json` prints one document, `{ "schema_version", "crawl" }`, where
+`crawl` is the crawl as the service returns it (or `null` when no crawl of the
+capture could be started); `haystack schema verify` prints its schema. Exit codes
+follow `haystack case-batch status`: 0 when the crawl finished (the bugs it
+found are in the output) or is still running under `--no-wait`; 2 when it ended
+without finishing (stopped early, or cancelled because a newer stop in the
+repository replaced it) or the machines it used could not be proven shut down;
+1 when the command failed or no crawl could be started.
+
+**The stop hook.** `haystack hooks install-session --cli claude` installs a
+Claude Code Stop hook that runs `haystack verify precompute --hook` whenever an
+agent turn stops. It captures the checkout within five seconds and hands the
+request to a detached sender; the service then starts the change's analysis
+and, for a repository set up for crawling, its crawl in the background, so
+`haystack verify` usually finds the crawl already running or finished. A newer stop in the same repository replaces an
+older crawl that has not finished. `haystack verify precompute` (without
+`--hook`) submits the same capture in the foreground and prints one line for
+the crawl it started.
+
+The hosted fleet run of exact pushed commits stays available:
 
 ```bash
+haystack verify hosted start owner/repo --base <sha> --head <sha> --json
 haystack verify hosted status cv_<48-lowercase-hex-characters> --wait --json
 haystack verify history owner/repo --limit 20
 ```
 
-Use the returned account-bound `next_command` verbatim on machines with multiple
-saved accounts. `haystack verify hosted start owner/repo --base <sha> --head
-<sha>` starts the same fleet execution through an explicit subcommand.
+Intent is optional there: `--intent-file <path>` accepts JSON with exactly
+`problem`, `goal`, and `intended_outcomes`. It waits up to 35 minutes by
+default; pass `--no-wait` to return after the fleet run is queued. Repeating the
+same repository, commits, and intent reuses the same run; choose a new
+`--idempotency-key` when a fresh execution is intentional. A wait that expires
+prints `timed_out: true` and `next_command`, then exits 2. Use the returned
+account-bound `next_command` verbatim on machines with multiple saved accounts.
 
 Retained fleet cases remain available through their exact case IDs:
 
@@ -230,8 +272,8 @@ haystack submit --no-wait            # Don't wait for analysis results
 ```
 
 **Pre-PR triage**: two checkers run in parallel, each one structured call to
-`gpt-6-astra` over the OpenAI Responses API at reasoning effort `max` (the
-highest the API accepts for that model). The key comes from `OPENAI_API_KEY`
+`gpt-6-astra` over the OpenAI Responses API at reasoning effort `xhigh`.
+The key comes from `OPENAI_API_KEY`
 if set, else from the SSM SecureString `/haystack/secrets/shared/prod/OPENAI_API_KEY`
 (us-west-2) through the default AWS credential chain (e.g. `AWS_PROFILE=prod`).
 With neither, each checker reports `no OpenAI key (...)` and submit continues.
@@ -462,7 +504,8 @@ haystack hooks install --force       # Overwrite existing hooks
 # Status
 haystack hooks status                # Check installation status
 
-# Session hooks (triage on CLI start)
+# Session hooks (triage on CLI start; Claude Code's Stop hook starts a crawl,
+# see `haystack verify`)
 haystack hooks install-session              # Auto-detect CLIs
 haystack hooks install-session --cli claude # Claude Code only
 haystack hooks install-session --cli all    # All detected CLIs
@@ -520,6 +563,145 @@ behind an `UNKNOWN` that is not an arrival or wall outcome, and behind an
 is `null` when the outcome has none. A case in `--cases` may carry `repeatIndex` (1-16), which
 must be part of its `contentDigest`, to run as a repeat next to its original.
 
+### `haystack db profile`
+
+Describe your production Postgres database so Haystack can build a full-size
+stand-in for testing, without any personal data leaving your environment. You
+run the profiler yourself, inside your own network, against your own database;
+it writes a JSON file you read before you send it.
+
+```bash
+# The connection string is read from the variable you name, never from a flag.
+export DATABASE_URL='postgres://profiler_readonly@db.internal:5432/app'
+haystack db profile --url-env DATABASE_URL --owner-table public.users --out profile.json
+
+# Read exactly what would be sent, in plain English.
+haystack db profile show profile.json
+
+# Send it for one repository.
+haystack db profile upload profile.json --repo acme/app
+```
+
+**Which database it is used for.** Uploaded without `--dependency`, the profile
+belongs to the repository, and Haystack builds the stand-in from it for the
+repository's one Postgres database, whatever id onboarding gave it. When the
+repository has several (or none), `haystack verify hosted start` says the
+stand-in was not used, and why, naming the ids; upload again with `--dependency <id>` to pick
+one. A profile uploaded with `--dependency <id>` is used for that database; when
+both apply, the newer upload is used. `haystack verify hosted start` prints one
+line saying which profile the stand-in was built from, or why it was not used, for every
+repository that has uploaded a profile.
+
+**Connection strings.** Any `postgres://` or `postgresql://` URL that
+node-postgres accepts works, including a Unix socket
+(`postgresql://profiler_readonly@/app?host=/var/run/postgresql`).
+
+**What it does to your database.** It only reads. Every query is a single
+`SELECT` in its own `READ ONLY` transaction (the session default is read only
+too, and the transaction's read-only state is checked before each query), with
+a statement timeout and a 5-second lock timeout so it never queues behind a
+migration. Each transaction is rolled back. All of them read one `REPEATABLE
+READ` snapshot, so every number describes the same moment even while your
+application writes; one extra connection holds that snapshot open for the run
+(on a primary, rows deleted during the run are cleaned up by vacuum only after
+it ends). A role with `SELECT` on the
+application tables is enough, and is what we recommend; tables with row-level
+security active for that role, or that the role cannot read, stop the run with
+their names. It works on Postgres 12 and later (tested on 16).
+
+**Run it on a read replica.** It works on a streaming-replication hot standby
+(tested on Postgres 16): nothing it runs writes, takes a write lock or creates a
+temporary table, and each transaction names its isolation level (`REPEATABLE
+READ`), so a replica whose default is `serializable` still answers. A
+replica does not count dead rows, so a profile taken there marks each table's
+dead rows as unavailable and says why; everything else is the same as on the
+primary. A replica may cancel a long query when replaying the primary's changes
+cannot wait for it (`canceling statement due to conflict with recovery`), and
+the run's snapshot is held for its whole length; the run then stops and names
+the fix: turn on `hot_standby_feedback` on the replica, or raise its
+`max_standby_streaming_delay` above `--statement-timeout`.
+
+**What leaves your database.** The database computes every count and pattern;
+no row ever reaches the profiler. A column's actual values are written only when
+all three hold:
+
+1. the column is a category: at most `--category-max-distinct` distinct values,
+   counted exactly over the whole table even when the table is otherwise
+   sampled (never estimated, so a column with many rare values cannot pass);
+2. the column is not personal. Personal means names, emails, phones, addresses,
+   free text (notes, comments, descriptions, messages), secrets, tokens and
+   passwords, IP and network addresses, dates of birth and government IDs. It is
+   decided from the column's name, from its type (`inet`, `cidr`, `macaddr`), and
+   from value shapes the database measures (the share of values shaped like an
+   email, phone number, IP address, secret, SSN, card number or "First Last"
+   name, and the number of words per value). When in doubt a column is personal,
+   and the profile records why;
+3. each value is shared by at least `--category-min-owners` owners. Owners are the
+   distinct rows of `--owner-table` (your users or organizations) reached by
+   following foreign keys from the column's table, and only owners that exist
+   there count (a key left dangling under a `NOT VALID` foreign key owns
+   nothing). A table with no foreign-key path to the owner table keeps no
+   values, shapes, JSON keys, percentiles or shares of true. Without
+   `--owner-table`, each row counts as an owner and the profile says so; a value
+   one heavy user repeats in thousands of rows then passes, so name an owner
+   table whenever you have one.
+
+Values your schema declares (enum labels, `CHECK (col IN (...))` lists) need no
+data and are kept with their share of rows. Everything else leaves only as
+numbers: row counts (and whether each was counted exactly or is the planner's
+estimate), how each table was read (in full, or the sample percentage and
+seed), table/index/TOAST sizes, dead-row share (or why it is unavailable), null
+share, distinct count, average width, the share of `true` in each boolean
+column, and the 5th/25th/50th/75th/95th percentiles of lengths and numbers
+(never the minimum or maximum; percentiles and the share of `true` are withheld
+for a column with fewer than `--category-min-owners` owners, and number
+percentiles are withheld for personal columns, because a percentile of birth
+dates is a birth date). Character shapes (`Aa Aa`, `+9 (9) 9-9`) and JSON key
+paths leave only when shared by at least `--category-min-owners` owners, and a
+JSON key path never leaves when any key on it looks like an email, phone number,
+IP address, SSN, card number or secret (`15551234567.verified` stays home). Odd
+rows (wrong type, oversize, bad encoding, rare nulls, rare shapes, missing JSON
+keys) are counted and described without their values, even for a single row.
+Index definitions are sent with every literal replaced by `?`, since a partial
+index can name a value (`WHERE email <> 'ceo@example.com'` is sent as `WHERE
+(email <> ?::text)`); the column names, operators and types stay. Settings are
+an allowlist of planner and storage settings; connection, authentication, SSL
+and file settings are never read.
+
+`show` prints every string the file carries. `upload` refuses a file with any
+field `show` does not print, a file whose percentiles include anything but
+p5 to p95, and a file with a literal left in an index definition. Haystack's
+server checks the file again and also refuses one made with a looser rule than
+at most 50 distinct values and at least 50 owners, a kept value its row share
+shows fewer than `--category-min-owners` rows hold, and any kept value shaped
+like an email, phone number, IP address, SSN, card number, token or password
+hash, whatever its column's type, or that does not fit its column's declared
+type (an integer column keeps only integers, a boolean column `true` or
+`false`, a date or time column dates or times; a column of any other type than
+text, numbers, booleans, dates and times keeps no data values, and enum labels
+leave as schema values). The profiler never writes such a value itself:
+however many owners share it (a date like `2031-07-19` has a phone number's
+shape, and so does any number of seven or more digits), it is withheld like a
+value too few owners share. An accepted file is stored once, under its
+SHA-256, in Haystack's results storage.
+
+Knobs and what turning them does:
+
+| Flag | Default | Turning it up | Turning it down |
+|------|---------|---------------|-----------------|
+| `--owner-table <schema.table>` | none (rows are owners) | n/a | n/a; naming one makes the owner threshold count people or organizations instead of rows, which is stricter and what we recommend. |
+| `--schema <name>` (repeatable) | every non-system schema | n/a | Profiles fewer schemas; foreign-key paths to the owner table only run through profiled tables. |
+| `--category-max-distinct <n>` | 50 (1 to 1000) | More columns count as categories and can keep values (each still needs enough owners); `upload` accepts at most 50. | Fewer columns keep values; more leave only as numbers and shapes. |
+| `--category-min-owners <n>` | 50 (5 to 1,000,000) | Stricter: fewer values, shapes, JSON keys and percentiles pass. | More rare values pass; each one is shared by fewer people. Not below 5; `upload` requires at least 50. |
+| `--scan-max-mb <mb>` | 256 | Bigger tables are read in full: exact counts and more rare-but-common-enough values, at the cost of more reading on your database. | Tables above this are read through `TABLESAMPLE SYSTEM` of about this size (same pages for every query). Owner counts in a sample are lower than the truth, so fewer values pass (never more). Row counts of sampled tables come from the planner's statistics, so a sampled table must have been analyzed; the profile records the sample percentage and seed and marks the row count as an estimate. Distinct counts of a sampled table are estimated from the sample (a single-column primary or unique key counts one per row, and a validated single-column foreign key counts the parent rows it references), except a column that could be a category: its distinct values are counted exactly, which reads that table in full once for all such columns (the run says which tables). |
+| `--concurrency <n>` | 2 (1 to 8) | Faster, more simultaneous load on your database. | Gentler; 1 runs one query at a time. Either way one more connection holds the run's snapshot. |
+| `--statement-timeout <seconds>` | 120 | Allows longer queries on big tables. | Caps each query sooner; a query that hits it stops the run and names the table. |
+
+Every failure is one line naming what failed. Nothing is written unless the
+whole profile succeeds. On a primary, dead-row shares come from
+`pg_stat_user_tables`; a table with rows but no counts there (statistics reset)
+stops the run and asks for `ANALYZE` on that table.
+
 ### `haystack policy`
 
 Manage review policies (`.haystack/review-policy.md`):

package/dist/commands/case-batch-contract.js

@@ -162,8 +162,8 @@ export function integer(value, what, minimum, maximum) {
     }
     return value;
 }
-/** Bounded JSON, as the combination-search hook already bounds its inputs:
- * finite numbers, no prototype-polluting keys, depth 64, 8 MiB serialized. */
+/** Bounded JSON: finite numbers, no prototype-polluting keys, depth 64, 8 MiB
+ * serialized. */
 export function boundedCaseBatchJson(value, what, maximumBytes = CASE_BATCH_MAX_INPUT_BYTES) {
     const visit = (item, depth) => {
         if (depth > 64)

package/dist/commands/case-batch.js

@@ -19,19 +19,8 @@ import { resolveAuthContext } from '../utils/auth.js';
 import { classifyHttpError, HaystackApiError, haystackApiUrl } from '../utils/haystack-api.js';
 import { boundedCaseBatchJson, buildCaseBatchRequest, CASE_BATCH_MAX_CASES, CASE_BATCH_MAX_CONCURRENT_CASES, CASE_BATCH_MAX_INPUT_BYTES, CASE_BATCH_MAX_CASE_WALL_MS, CASE_BATCH_MAX_TOTAL_BUDGET_MS, CASE_BATCH_MIN_CASE_WALL_MS, CASE_BATCH_MIN_TOTAL_BUDGET_MS, CaseBatchRequestValidationError, CaseBatchResponseError, isJsonObject, isTerminalCaseBatchStatus, parseCaseBatchCommit, parseCaseBatchIdempotencyKey, parseCaseBatchLimits, parseCaseBatchRepository, parseCaseBatchSnapshot, parseCaseBatchSource, parseCaseBatchWorld, parseProductCases, } from './case-batch-contract.js';
 /** The CLI gateway mount. The same worker handlers are `/v1/case-batches` and
- * `/api/cloud-verifier/case-batches`; CLI callers use the agent gateway, as
- * they already do for `/api/agent/cloud-verifier/searches`.
- *
- * BLOCKED ON THE SERVER LANE. No handler serves this path in this repository
- * yet, and the auth worker does not proxy it: `isSearchProxyPath` in
- * `infra/auth-worker/index.js` covers `/api/agent/cloud-verifier/searches`
- * only, and `agent/cloudflare/src/combination-search-route.ts` has no
- * case-batch sibling. Until Lane A of `docs/case-batch-coordinator.md` lands
- * `POST/GET /v1/case-batches`, `GET/DELETE /v1/case-batches/:runId`,
- * `GET /v1/case-batches/:runId/bundle[/path]` and the matching auth-worker
- * proxy entry, every command in this file reaches a 404 in production. That
- * lane must merge first; this file is written against the frozen contract
- * ahead of it, which is what the implementation plan assigns to Lane D.
+ * `/api/cloud-verifier/case-batches` (`agent/cloudflare/src/case-batch-route.ts`);
+ * CLI callers use the agent gateway, which `infra/auth-worker/index.js` proxies.
  *
  * Three things the server side must hold for this file to work as written:
  * the 8 MiB request bound, `repository`-scoped reads with `limit` and
@@ -40,8 +29,8 @@ import { boundedCaseBatchJson, buildCaseBatchRequest, CASE_BATCH_MAX_CASES, CASE
  * is the one artifact with no parent digest to check it against). */
 const GATEWAY = '/api/agent/cloud-verifier/case-batches';
 const RUN_ID = /^cv_[0-9a-f]{48}$/;
-/** Page size for the bounded case pagination; the search read route caps
- * `limit` at 500 and the batch read mirrors it. */
+/** Page size for the bounded case pagination; the batch read route caps
+ * `limit` at 500. */
 const CASE_PAGE_LIMIT = 500;
 /** A hard stop on pagination independent of what the server reports, so a
  * cursor that never terminates cannot spin the CLI forever. */
@@ -102,8 +91,9 @@ function requireRunId(runId) {
     return runId;
 }
 /** Every outbound request. The header set is built here and nowhere else, so
- * the credential surface of this command is one line: the login bearer. */
-async function gatewayFetch(path, token, init = { method: 'GET' }) {
+ * the credential surface of this command is one line: the login bearer.
+ * `haystack verify` reads crawls through this same client. */
+export async function gatewayFetch(path, token, init = { method: 'GET' }) {
     const headers = new Headers();
     headers.set('Authorization', `Bearer ${token}`);
     headers.set('Accept', init.accept ?? 'application/json');

package/dist/commands/crawl-contract.js

@@ -0,0 +1,3 @@
+export const CRAWL_TITLE_MAX_CHARS = 200;
+export const CRAWL_MAX_FINDINGS = 50;
+export const CRAWL_MAX_FINDING_STEPS = 64;