@haystackeditor/cli 0.15.18 → 0.15.19
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/assets/skills/map-your-system.md +138 -0
- package/dist/commands/review.d.ts +24 -0
- package/dist/commands/review.js +193 -0
- package/dist/commands/skills.js +4 -0
- package/dist/index.js +29 -0
- package/dist/utils/analysis-api.d.ts +33 -0
- package/dist/utils/analysis-api.js +108 -3
- package/package.json +1 -1
|
@@ -0,0 +1,138 @@
|
|
|
1
|
+
# Map Your System for Haystack QA
|
|
2
|
+
|
|
3
|
+
Create `.haystack/system.yml` — a short file of facts about how this
|
|
4
|
+
repository's system works, so Haystack QA can run it.
|
|
5
|
+
|
|
6
|
+
## Why this file exists
|
|
7
|
+
|
|
8
|
+
When Haystack reviews a risky PR, it doesn't just read the diff — it runs the
|
|
9
|
+
affected code in a sandbox and captures evidence. Haystack decides **what** to
|
|
10
|
+
test from each PR on its own. What it can't always infer is **how your system
|
|
11
|
+
works**: how to install and build it, how services boot, how to know they're
|
|
12
|
+
ready, how to seed data, how to act as a logged-in test user. That is what
|
|
13
|
+
this file provides. Every fact you add converts risks Haystack identified but
|
|
14
|
+
couldn't execute into checks that actually run.
|
|
15
|
+
|
|
16
|
+
## Hard rules
|
|
17
|
+
|
|
18
|
+
1. **Facts only — never test selection.** The file describes how the system
|
|
19
|
+
works. Keys like `scenarios`, `tests`, `skip`, or `focus` are rejected;
|
|
20
|
+
Haystack alone decides what to test.
|
|
21
|
+
2. **Only verified facts.** Run every command yourself before declaring it.
|
|
22
|
+
A wrong fact is worse than a missing one — Haystack tracks which declared
|
|
23
|
+
facts fail and will flag them on PRs.
|
|
24
|
+
3. **No secrets.** Reference environment variable *names* if needed, never
|
|
25
|
+
values. Anything that looks like a credential is rejected.
|
|
26
|
+
4. **Personas are seeded FAKE users** reachable through dev/test-only login
|
|
27
|
+
paths. Never real accounts, never customer data. If a dev login helper
|
|
28
|
+
exists, it must be disabled in production builds.
|
|
29
|
+
5. **Small and true beats big and aspirational.** Ten verified lines are more
|
|
30
|
+
useful than sixty guesses. Omit anything you could not verify.
|
|
31
|
+
|
|
32
|
+
## Steps
|
|
33
|
+
|
|
34
|
+
### 1. Inspect the repository
|
|
35
|
+
|
|
36
|
+
Determine, with evidence from the tree (lockfiles, manifests, scripts,
|
|
37
|
+
docker-compose, CI config, READMEs):
|
|
38
|
+
|
|
39
|
+
- package manager, install command, and any required build/codegen steps
|
|
40
|
+
- each runnable service: start command, port, and how "ready" is observable
|
|
41
|
+
(health endpoint, log line, or a command that exits 0)
|
|
42
|
+
- how to seed and reset local/test data
|
|
43
|
+
- how existing tests authenticate — is there a seeded test user or dev login?
|
|
44
|
+
- the test runner and any test helpers/factories generated tests should reuse
|
|
45
|
+
- external services the code calls, and how development runs without hitting
|
|
46
|
+
them for real (mock fixture, log driver, sandbox mode)
|
|
47
|
+
- where runtime logs are written
|
|
48
|
+
|
|
49
|
+
### 2. Verify each fact by running it
|
|
50
|
+
|
|
51
|
+
From a clean state where possible: run the install, run the build, boot each
|
|
52
|
+
service and confirm its readiness signal, run the seed command, run the login
|
|
53
|
+
helper. Record the exact commands that worked. Drop anything you could not
|
|
54
|
+
make work — do not declare it with a caveat.
|
|
55
|
+
|
|
56
|
+
### 3. Ask the developer only what you cannot determine
|
|
57
|
+
|
|
58
|
+
Typical questions worth asking a human: "Is there a dev-only way to log in as
|
|
59
|
+
a test user?", "Which services must be running for the API to work?", "Is
|
|
60
|
+
there a canonical seed script?". Keep it to the few facts you genuinely
|
|
61
|
+
cannot infer or verify.
|
|
62
|
+
|
|
63
|
+
### 4. Write `.haystack/system.yml`
|
|
64
|
+
|
|
65
|
+
```yaml
|
|
66
|
+
version: 1
|
|
67
|
+
|
|
68
|
+
environment:
|
|
69
|
+
setup:
|
|
70
|
+
- run: "pnpm install --frozen-lockfile"
|
|
71
|
+
- run: "pnpm codegen"
|
|
72
|
+
rationale: "generated types are imported throughout"
|
|
73
|
+
toolchain: { node: "20", pnpm: "10" }
|
|
74
|
+
|
|
75
|
+
services:
|
|
76
|
+
api:
|
|
77
|
+
root: "apps/api" # optional
|
|
78
|
+
boot: "pnpm dev"
|
|
79
|
+
ready: { http: "http://localhost:8080/health" } # or { log_pattern: "listening" } or { command: "pg_isready" }
|
|
80
|
+
env: { APP_ENV: "test" }
|
|
81
|
+
depends_on: [postgres]
|
|
82
|
+
|
|
83
|
+
data:
|
|
84
|
+
seed: "pnpm db:seed --profile=qa"
|
|
85
|
+
reset: "pnpm db:reset"
|
|
86
|
+
personas:
|
|
87
|
+
admin:
|
|
88
|
+
login: "pnpm test:login admin"
|
|
89
|
+
yields: "prints an Authorization header on stdout"
|
|
90
|
+
|
|
91
|
+
observability:
|
|
92
|
+
logs: ["apps/api/log/dev.log"]
|
|
93
|
+
inspect_db: "pnpm db:console --readonly"
|
|
94
|
+
|
|
95
|
+
external_services:
|
|
96
|
+
stripe:
|
|
97
|
+
offline: "repo ships a stripe-mock fixture used by tests"
|
|
98
|
+
must_not_touch_live: true
|
|
99
|
+
|
|
100
|
+
testing:
|
|
101
|
+
runner: "pnpm vitest"
|
|
102
|
+
probe_helpers: "tests/helpers/ has request builders and DB factories"
|
|
103
|
+
|
|
104
|
+
constraints:
|
|
105
|
+
- "full build takes ~6 min; apps/api boots standalone without the web build"
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
Every section is optional — include only what you verified. All commands run
|
|
109
|
+
from the repo root unless a `cwd`/`root` says otherwise.
|
|
110
|
+
|
|
111
|
+
### 5. Self-check before opening the PR
|
|
112
|
+
|
|
113
|
+
- [ ] `version: 1` is present and the file is valid YAML
|
|
114
|
+
- [ ] no test-selection keys (`scenarios`, `tests`, `skip`, `focus`, ...)
|
|
115
|
+
- [ ] no secret values anywhere (env var names only)
|
|
116
|
+
- [ ] every `run`/`boot`/`login`/`seed` command was executed and worked
|
|
117
|
+
- [ ] every service has a `ready` signal you observed
|
|
118
|
+
- [ ] personas are seeded fake users via dev/test-only paths
|
|
119
|
+
|
|
120
|
+
Haystack re-validates the file on its next QA run and reports precise errors
|
|
121
|
+
if something is wrong.
|
|
122
|
+
|
|
123
|
+
### 6. Open a small, reviewable PR
|
|
124
|
+
|
|
125
|
+
Add only `.haystack/system.yml` (plus a dev login helper if one was genuinely
|
|
126
|
+
needed — dev/test-gated, with a test proving it's off in production). In the
|
|
127
|
+
PR description, list which facts you verified by running them and which came
|
|
128
|
+
from the developer's answers.
|
|
129
|
+
|
|
130
|
+
## What happens next
|
|
131
|
+
|
|
132
|
+
Haystack treats each entry as a claim. When a QA run executes a declared
|
|
133
|
+
command successfully, the fact is marked **verified** with the run's receipt;
|
|
134
|
+
if it fails at a newer commit it is marked **broken**, and Haystack will ask
|
|
135
|
+
about it on a PR instead of trusting it. When QA hits something the map
|
|
136
|
+
doesn't cover, it asks at most a couple of targeted questions on the affected
|
|
137
|
+
PR, each with a ready-to-paste snippet — answering them grows this file over
|
|
138
|
+
time.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* review command — Trigger a fresh Haystack analysis for a PR.
|
|
3
|
+
*
|
|
4
|
+
* haystack review [pr] [--no-wait]
|
|
5
|
+
*
|
|
6
|
+
* Haystack runs a full analysis when a PR is opened (or marked ready);
|
|
7
|
+
* later pushes only get a resolution check against the existing findings.
|
|
8
|
+
* This command is the explicit escape hatch: it re-runs full detection on
|
|
9
|
+
* the PR's current head — use it after substantial new commits, or any
|
|
10
|
+
* time you want a fresh second opinion.
|
|
11
|
+
*
|
|
12
|
+
* When called without a PR identifier, reads the last submitted PR from
|
|
13
|
+
* `.haystack/pending-submit.json` (written by `haystack submit`).
|
|
14
|
+
*
|
|
15
|
+
* Accepts a PR identifier in several formats:
|
|
16
|
+
* haystack review # Last submitted PR
|
|
17
|
+
* haystack review 123 # Uses current repo's owner/repo
|
|
18
|
+
* haystack review owner/repo#123 # Fully qualified
|
|
19
|
+
* haystack review https://github.com/owner/repo/pull/123
|
|
20
|
+
*/
|
|
21
|
+
export interface ReviewOptions {
|
|
22
|
+
wait?: boolean;
|
|
23
|
+
}
|
|
24
|
+
export declare function reviewCommand(identifier: string | undefined, options: ReviewOptions): Promise<void>;
|
|
@@ -0,0 +1,193 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* review command — Trigger a fresh Haystack analysis for a PR.
|
|
3
|
+
*
|
|
4
|
+
* haystack review [pr] [--no-wait]
|
|
5
|
+
*
|
|
6
|
+
* Haystack runs a full analysis when a PR is opened (or marked ready);
|
|
7
|
+
* later pushes only get a resolution check against the existing findings.
|
|
8
|
+
* This command is the explicit escape hatch: it re-runs full detection on
|
|
9
|
+
* the PR's current head — use it after substantial new commits, or any
|
|
10
|
+
* time you want a fresh second opinion.
|
|
11
|
+
*
|
|
12
|
+
* When called without a PR identifier, reads the last submitted PR from
|
|
13
|
+
* `.haystack/pending-submit.json` (written by `haystack submit`).
|
|
14
|
+
*
|
|
15
|
+
* Accepts a PR identifier in several formats:
|
|
16
|
+
* haystack review # Last submitted PR
|
|
17
|
+
* haystack review 123 # Uses current repo's owner/repo
|
|
18
|
+
* haystack review owner/repo#123 # Fully qualified
|
|
19
|
+
* haystack review https://github.com/owner/repo/pull/123
|
|
20
|
+
*/
|
|
21
|
+
import chalk from 'chalk';
|
|
22
|
+
import { triggerAnalysis, fetchTriggerStatus } from '../utils/analysis-api.js';
|
|
23
|
+
import { fetchTriageHeadSha } from './triage.js';
|
|
24
|
+
import { parseRemoteUrl } from '../utils/git.js';
|
|
25
|
+
import { loadToken } from '../utils/auth.js';
|
|
26
|
+
import { loadPendingSubmit } from '../utils/pending-state.js';
|
|
27
|
+
import { triageCommand } from './triage.js';
|
|
28
|
+
// ============================================================================
|
|
29
|
+
// Constants
|
|
30
|
+
// ============================================================================
|
|
31
|
+
const STATUS_POLL_INTERVAL_MS = 10_000;
|
|
32
|
+
// Full analyses regularly take 10+ minutes on large PRs; this bounds the
|
|
33
|
+
// CLI's wait, not the analysis itself (results land on the feed regardless).
|
|
34
|
+
const STATUS_POLL_TIMEOUT_MS = 25 * 60 * 1000;
|
|
35
|
+
// ============================================================================
|
|
36
|
+
// PR identifier parsing (shared pattern with request-review.ts / triage.ts)
|
|
37
|
+
// ============================================================================
|
|
38
|
+
function parsePRIdentifier(identifier) {
|
|
39
|
+
const urlMatch = identifier.match(/^https?:\/\/github\.com\/([^/]+)\/([^/]+)\/pull\/(\d+)\/?$/);
|
|
40
|
+
if (urlMatch) {
|
|
41
|
+
return { owner: urlMatch[1], repo: urlMatch[2], prNumber: parseInt(urlMatch[3], 10) };
|
|
42
|
+
}
|
|
43
|
+
const qualifiedMatch = identifier.match(/^([^/]+)\/([^#]+)#(\d+)$/);
|
|
44
|
+
if (qualifiedMatch) {
|
|
45
|
+
return { owner: qualifiedMatch[1], repo: qualifiedMatch[2], prNumber: parseInt(qualifiedMatch[3], 10) };
|
|
46
|
+
}
|
|
47
|
+
const numberMatch = identifier.match(/^#?(\d+)$/);
|
|
48
|
+
if (numberMatch) {
|
|
49
|
+
const prNumber = parseInt(numberMatch[1], 10);
|
|
50
|
+
try {
|
|
51
|
+
const remote = parseRemoteUrl('origin');
|
|
52
|
+
return { owner: remote.owner, repo: remote.repo, prNumber };
|
|
53
|
+
}
|
|
54
|
+
catch {
|
|
55
|
+
throw new Error(`Cannot determine repository for PR #${prNumber}.\n` +
|
|
56
|
+
`Use the full format: haystack review owner/repo#${prNumber}`);
|
|
57
|
+
}
|
|
58
|
+
}
|
|
59
|
+
throw new Error(`Invalid PR identifier: "${identifier}"\n\n` +
|
|
60
|
+
`Accepted formats:\n` +
|
|
61
|
+
` haystack review 123\n` +
|
|
62
|
+
` haystack review owner/repo#123\n` +
|
|
63
|
+
` haystack review https://github.com/owner/repo/pull/123`);
|
|
64
|
+
}
|
|
65
|
+
function resolvePR(identifier) {
|
|
66
|
+
if (identifier) {
|
|
67
|
+
try {
|
|
68
|
+
return parsePRIdentifier(identifier);
|
|
69
|
+
}
|
|
70
|
+
catch (err) {
|
|
71
|
+
console.error(chalk.red(`\n${err.message}\n`));
|
|
72
|
+
process.exit(1);
|
|
73
|
+
}
|
|
74
|
+
}
|
|
75
|
+
const pending = loadPendingSubmit();
|
|
76
|
+
if (!pending) {
|
|
77
|
+
console.log(chalk.dim('No pending submits. Pass a PR number to review a specific PR.\n'));
|
|
78
|
+
return null;
|
|
79
|
+
}
|
|
80
|
+
return { owner: pending.owner, repo: pending.repo, prNumber: pending.prNumber };
|
|
81
|
+
}
|
|
82
|
+
// ============================================================================
|
|
83
|
+
// Waiting
|
|
84
|
+
// ============================================================================
|
|
85
|
+
async function waitForRun(requestId, token) {
|
|
86
|
+
const deadline = Date.now() + STATUS_POLL_TIMEOUT_MS;
|
|
87
|
+
let consecutiveErrors = 0;
|
|
88
|
+
while (Date.now() < deadline) {
|
|
89
|
+
const result = await fetchTriggerStatus(requestId, token);
|
|
90
|
+
if (result.status === 'COMPLETED')
|
|
91
|
+
return 'completed';
|
|
92
|
+
if (result.status === 'FAILED')
|
|
93
|
+
return 'failed';
|
|
94
|
+
if (result.status === 'SUPERSEDED')
|
|
95
|
+
return 'superseded';
|
|
96
|
+
if (result.status === 'error') {
|
|
97
|
+
consecutiveErrors++;
|
|
98
|
+
if (consecutiveErrors >= 5)
|
|
99
|
+
return 'failed';
|
|
100
|
+
}
|
|
101
|
+
else {
|
|
102
|
+
consecutiveErrors = 0;
|
|
103
|
+
}
|
|
104
|
+
process.stderr.write(`\r ${chalk.dim('Analysis in progress...')}${' '.repeat(5)}`);
|
|
105
|
+
// Deliberately NOT unref'd: during the wait this timer can be the only
|
|
106
|
+
// live handle, and an unref'd timer lets Node exit mid-wait.
|
|
107
|
+
await new Promise((resolve) => {
|
|
108
|
+
setTimeout(resolve, STATUS_POLL_INTERVAL_MS);
|
|
109
|
+
});
|
|
110
|
+
}
|
|
111
|
+
return 'timeout';
|
|
112
|
+
}
|
|
113
|
+
// ============================================================================
|
|
114
|
+
// Main command
|
|
115
|
+
// ============================================================================
|
|
116
|
+
export async function reviewCommand(identifier, options) {
|
|
117
|
+
const pr = resolvePR(identifier);
|
|
118
|
+
if (!pr)
|
|
119
|
+
return;
|
|
120
|
+
const prLabel = `${pr.owner}/${pr.repo}#${pr.prNumber}`;
|
|
121
|
+
let token = null;
|
|
122
|
+
try {
|
|
123
|
+
token = await loadToken({ owner: pr.owner, repo: pr.repo });
|
|
124
|
+
}
|
|
125
|
+
catch (err) {
|
|
126
|
+
console.error(chalk.red(`\nCould not load Haystack credentials: ${err.message}\n`));
|
|
127
|
+
process.exit(1);
|
|
128
|
+
}
|
|
129
|
+
if (!token) {
|
|
130
|
+
console.error(chalk.red('\nTriggering an analysis requires authentication.'));
|
|
131
|
+
console.log(chalk.dim(' Run ') + chalk.cyan('haystack login') + chalk.dim(' first.\n'));
|
|
132
|
+
process.exit(1);
|
|
133
|
+
}
|
|
134
|
+
console.log(chalk.dim(`\nRequesting a fresh analysis for ${prLabel}...`));
|
|
135
|
+
// Resolve the current head so ingestion's dedup compares commits — an
|
|
136
|
+
// in-flight run for an OLDER head must be superseded, not reused. A
|
|
137
|
+
// failed metadata fetch degrades to a head-less trigger (the dedup is a
|
|
138
|
+
// freshness aid, not a correctness requirement) instead of crashing.
|
|
139
|
+
let headSha;
|
|
140
|
+
try {
|
|
141
|
+
headSha = await fetchTriageHeadSha(pr, token) ?? undefined;
|
|
142
|
+
}
|
|
143
|
+
catch (err) {
|
|
144
|
+
console.log(chalk.dim(` Could not resolve the PR head (${err.message}) — continuing without it.`));
|
|
145
|
+
}
|
|
146
|
+
const trigger = await triggerAnalysis(pr.owner, pr.repo, pr.prNumber, token, headSha);
|
|
147
|
+
if (trigger.status === 'rate_limited') {
|
|
148
|
+
console.error(chalk.red(`\n ${trigger.message}`));
|
|
149
|
+
if (trigger.retryAfterSeconds) {
|
|
150
|
+
console.log(chalk.dim(` Try again in ~${Math.ceil(trigger.retryAfterSeconds / 60)} minute(s).\n`));
|
|
151
|
+
}
|
|
152
|
+
process.exit(1);
|
|
153
|
+
}
|
|
154
|
+
if (trigger.status === 'denied') {
|
|
155
|
+
console.error(chalk.red(`\n ${trigger.message}\n`));
|
|
156
|
+
process.exit(1);
|
|
157
|
+
}
|
|
158
|
+
if (trigger.status === 'error') {
|
|
159
|
+
console.error(chalk.red(`\n Failed to trigger analysis: ${trigger.message}\n`));
|
|
160
|
+
process.exit(1);
|
|
161
|
+
}
|
|
162
|
+
if (trigger.status === 'already_processing') {
|
|
163
|
+
console.log(chalk.yellow(' An analysis for this PR is already running — waiting on it instead.'));
|
|
164
|
+
}
|
|
165
|
+
else {
|
|
166
|
+
console.log(chalk.green(' Analysis started.'));
|
|
167
|
+
}
|
|
168
|
+
if (options.wait === false) {
|
|
169
|
+
console.log(chalk.dim(' Check results later with: ') + chalk.cyan(`haystack triage ${prLabel}`));
|
|
170
|
+
console.log(chalk.dim(' Or view the review: ') + chalk.cyan(`https://haystackeditor.com/review/${pr.owner}/${pr.repo}/${pr.prNumber}\n`));
|
|
171
|
+
return;
|
|
172
|
+
}
|
|
173
|
+
const outcome = await waitForRun(trigger.requestId, token);
|
|
174
|
+
process.stderr.write('\r' + ' '.repeat(60) + '\r');
|
|
175
|
+
if (outcome === 'failed') {
|
|
176
|
+
console.error(chalk.red('\n Analysis run failed.'));
|
|
177
|
+
console.log(chalk.dim(' Check the review page: ') + chalk.cyan(`https://haystackeditor.com/review/${pr.owner}/${pr.repo}/${pr.prNumber}\n`));
|
|
178
|
+
process.exit(1);
|
|
179
|
+
}
|
|
180
|
+
if (outcome === 'superseded') {
|
|
181
|
+
console.log(chalk.yellow('\n This run was superseded by a newer analysis (a push likely landed meanwhile).'));
|
|
182
|
+
console.log(chalk.dim(' Showing the latest results instead.'));
|
|
183
|
+
await triageCommand(prLabel, { wait: false });
|
|
184
|
+
return;
|
|
185
|
+
}
|
|
186
|
+
if (outcome === 'timeout') {
|
|
187
|
+
console.log(chalk.yellow('\n Still running after 25 minutes — giving up on waiting.'));
|
|
188
|
+
console.log(chalk.dim(' Check results later with: ') + chalk.cyan(`haystack triage ${prLabel}\n`));
|
|
189
|
+
return;
|
|
190
|
+
}
|
|
191
|
+
// Completed — render results through the triage view (no extra wait).
|
|
192
|
+
await triageCommand(prLabel, { wait: false });
|
|
193
|
+
}
|
package/dist/commands/skills.js
CHANGED
|
@@ -15,6 +15,7 @@ function getSkillsSourceDir() {
|
|
|
15
15
|
const SKILL_FILES = [
|
|
16
16
|
'setup-haystack.md',
|
|
17
17
|
'submit.md',
|
|
18
|
+
'map-your-system.md',
|
|
18
19
|
];
|
|
19
20
|
function installSkillFiles(targetDir) {
|
|
20
21
|
const sourceDir = getSkillsSourceDir();
|
|
@@ -55,6 +56,7 @@ export async function installSkills(options) {
|
|
|
55
56
|
}
|
|
56
57
|
const claudeCommandPath = join(claudeCommandsDir, 'setup-haystack.md');
|
|
57
58
|
writeFileSync(claudeCommandPath, '# Set Up Haystack\n\nFollow .agents/skills/setup-haystack.md to set up Haystack for this repo.\n');
|
|
59
|
+
writeFileSync(join(claudeCommandsDir, 'map-your-system.md'), '# Map Your System for Haystack QA\n\nFollow .agents/skills/map-your-system.md to create .haystack/system.yml — verified facts about how this system builds, boots, seeds data, and logs in, so Haystack QA can run it.\n');
|
|
58
60
|
showSuccessMessage();
|
|
59
61
|
}
|
|
60
62
|
function showSuccessMessage() {
|
|
@@ -62,6 +64,7 @@ function showSuccessMessage() {
|
|
|
62
64
|
console.log(chalk.white('Available skills:'));
|
|
63
65
|
console.log(chalk.cyan(' /setup-haystack ') + chalk.gray('- Create .haystack.json with AI assistance'));
|
|
64
66
|
console.log(chalk.cyan(' /submit ') + chalk.gray('- Submit a PR via Haystack'));
|
|
67
|
+
console.log(chalk.cyan(' /map-your-system ') + chalk.gray('- Write .haystack/system.yml so Haystack QA can run this system'));
|
|
65
68
|
console.log();
|
|
66
69
|
console.log(chalk.white('Run in your coding CLI:'));
|
|
67
70
|
console.log(chalk.cyan(' /setup-haystack'));
|
|
@@ -71,5 +74,6 @@ export async function listSkills() {
|
|
|
71
74
|
console.log(chalk.white('\nAvailable Haystack skills:\n'));
|
|
72
75
|
console.log(chalk.cyan(' /setup-haystack ') + chalk.gray('- Master setup - creates .haystack.json'));
|
|
73
76
|
console.log(chalk.cyan(' /submit ') + chalk.gray('- Submit a PR via Haystack'));
|
|
77
|
+
console.log(chalk.cyan(' /map-your-system ') + chalk.gray('- Write .haystack/system.yml so Haystack QA can run this system'));
|
|
74
78
|
console.log();
|
|
75
79
|
}
|
package/dist/index.js
CHANGED
|
@@ -36,6 +36,7 @@ import { listPolicies, addPolicy, removePolicy, initPolicies, addInstruction } f
|
|
|
36
36
|
import { triageCommand } from './commands/triage.js';
|
|
37
37
|
import { dismissCommand, markReviewedCommand, undismissCommand } from './commands/dismiss.js';
|
|
38
38
|
import { requestReviewCommand } from './commands/request-review.js';
|
|
39
|
+
import { reviewCommand } from './commands/review.js';
|
|
39
40
|
import { prStatusCommand } from './commands/pr-status.js';
|
|
40
41
|
import { prReadCommand } from './commands/pr.js';
|
|
41
42
|
import { inboxListCommand } from './commands/inbox.js';
|
|
@@ -449,6 +450,34 @@ Examples:
|
|
|
449
450
|
haystack undismiss acme/widgets#99 # Undo for specific repo
|
|
450
451
|
`)
|
|
451
452
|
.action(undismissCommand);
|
|
453
|
+
program
|
|
454
|
+
.command('review [pr]')
|
|
455
|
+
.description('Trigger a fresh Haystack analysis for a PR')
|
|
456
|
+
.option('--no-wait', 'Exit after triggering instead of waiting for results')
|
|
457
|
+
.addHelpText('after', `
|
|
458
|
+
Re-run full Haystack analysis on a PR's current head.
|
|
459
|
+
|
|
460
|
+
Haystack analyzes a PR once when it's opened (or marked ready); later
|
|
461
|
+
pushes only get a resolution check against the existing findings. Use
|
|
462
|
+
this command when new commits deserve fresh detection, or any time you
|
|
463
|
+
want a second opinion.
|
|
464
|
+
|
|
465
|
+
When called without a PR identifier, uses the last submitted PR.
|
|
466
|
+
|
|
467
|
+
PR identifier formats:
|
|
468
|
+
(none) Last submitted PR
|
|
469
|
+
123 PR number (uses current repo)
|
|
470
|
+
#123 PR number with hash
|
|
471
|
+
owner/repo#123 Fully qualified
|
|
472
|
+
https://github.com/owner/repo/pull/123 GitHub URL
|
|
473
|
+
|
|
474
|
+
Examples:
|
|
475
|
+
haystack review # Last submitted PR
|
|
476
|
+
haystack review 42 # Current repo, PR #42
|
|
477
|
+
haystack review acme/widgets#99 # Specific repo
|
|
478
|
+
haystack review 42 --no-wait # Trigger and exit
|
|
479
|
+
`)
|
|
480
|
+
.action(reviewCommand);
|
|
452
481
|
program
|
|
453
482
|
.command('request-review <pr> [reviewer]')
|
|
454
483
|
.description('Tag a PR as needing human review')
|
|
@@ -110,6 +110,39 @@ export type AnalysisReadyResult = {
|
|
|
110
110
|
status: 'error';
|
|
111
111
|
message: string;
|
|
112
112
|
};
|
|
113
|
+
export type TriggerAnalysisResult = {
|
|
114
|
+
status: 'started';
|
|
115
|
+
requestId: string;
|
|
116
|
+
} | {
|
|
117
|
+
status: 'already_processing';
|
|
118
|
+
requestId: string;
|
|
119
|
+
} | {
|
|
120
|
+
status: 'rate_limited';
|
|
121
|
+
message: string;
|
|
122
|
+
retryAfterSeconds?: number;
|
|
123
|
+
} | {
|
|
124
|
+
status: 'denied';
|
|
125
|
+
message: string;
|
|
126
|
+
} | {
|
|
127
|
+
status: 'error';
|
|
128
|
+
message: string;
|
|
129
|
+
};
|
|
130
|
+
/**
|
|
131
|
+
* Trigger a fresh full analysis for a PR via the auth-worker
|
|
132
|
+
* (`/api/analysis-trigger/{owner}/{repo}/{pr}`). Requires auth — triggers
|
|
133
|
+
* start a real analysis run. Returns the request id to wait on.
|
|
134
|
+
*/
|
|
135
|
+
export declare function triggerAnalysis(owner: string, repo: string, prNumber: number, token: string, headSha?: string): Promise<TriggerAnalysisResult>;
|
|
136
|
+
export type TriggerRunStatus = 'pending' | 'COMPLETED' | 'FAILED' | 'SUPERSEDED' | 'error';
|
|
137
|
+
/**
|
|
138
|
+
* Poll the status of a triggered run by request id. The PR-level result
|
|
139
|
+
* endpoint can't distinguish a fresh re-run on the same head from the
|
|
140
|
+
* previous completed analysis — the request id can.
|
|
141
|
+
*/
|
|
142
|
+
export declare function fetchTriggerStatus(requestId: string, token: string): Promise<{
|
|
143
|
+
status: TriggerRunStatus;
|
|
144
|
+
message?: string;
|
|
145
|
+
}>;
|
|
113
146
|
/**
|
|
114
147
|
* Check if analysis results are available for a PR by polling the result endpoint.
|
|
115
148
|
* Returns 'ready' if complete, 'pending' if not yet available, or 'error' for server failures.
|
|
@@ -61,9 +61,114 @@ function apiHeaders(token) {
|
|
|
61
61
|
}
|
|
62
62
|
return headers;
|
|
63
63
|
}
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
64
|
+
/**
|
|
65
|
+
* Trigger a fresh full analysis for a PR via the auth-worker
|
|
66
|
+
* (`/api/analysis-trigger/{owner}/{repo}/{pr}`). Requires auth — triggers
|
|
67
|
+
* start a real analysis run. Returns the request id to wait on.
|
|
68
|
+
*/
|
|
69
|
+
export async function triggerAnalysis(owner, repo, prNumber, token, headSha) {
|
|
70
|
+
const url = `${ANALYSIS_API_BASE}/api/analysis-trigger/${owner}/${repo}/${prNumber}`;
|
|
71
|
+
let response;
|
|
72
|
+
try {
|
|
73
|
+
// Force semantics ride on a server-set header at the trigger route.
|
|
74
|
+
// commitSha matters for dedup: without it, ingestion matches this
|
|
75
|
+
// request onto any in-flight run for an OLDER head and the CLI would
|
|
76
|
+
// wait on (and render) stale results.
|
|
77
|
+
response = await fetch(url, {
|
|
78
|
+
method: 'POST',
|
|
79
|
+
headers: { ...apiHeaders(token), 'Content-Type': 'application/json' },
|
|
80
|
+
body: JSON.stringify(headSha ? { commitSha: headSha } : {}),
|
|
81
|
+
});
|
|
82
|
+
}
|
|
83
|
+
catch (err) {
|
|
84
|
+
return { status: 'error', message: err.message };
|
|
85
|
+
}
|
|
86
|
+
let data = {};
|
|
87
|
+
let parseFailed = false;
|
|
88
|
+
try {
|
|
89
|
+
data = await response.json();
|
|
90
|
+
}
|
|
91
|
+
catch {
|
|
92
|
+
// Recorded, not swallowed: for non-OK responses the status-code handling
|
|
93
|
+
// below surfaces the failure; a 2xx with an unparseable body is its own
|
|
94
|
+
// explicit error (never "empty data") so callers can't mistake a
|
|
95
|
+
// malformed response for a normal one.
|
|
96
|
+
parseFailed = true;
|
|
97
|
+
}
|
|
98
|
+
if (parseFailed && response.ok) {
|
|
99
|
+
return { status: 'error', message: `Trigger endpoint returned invalid JSON (HTTP ${response.status})` };
|
|
100
|
+
}
|
|
101
|
+
if (response.status === 429) {
|
|
102
|
+
return {
|
|
103
|
+
status: 'rate_limited',
|
|
104
|
+
message: data.message || 'Too many analysis requests. Please wait before trying again.',
|
|
105
|
+
retryAfterSeconds: data.retryAfter,
|
|
106
|
+
};
|
|
107
|
+
}
|
|
108
|
+
if (response.status === 401 || response.status === 403) {
|
|
109
|
+
return {
|
|
110
|
+
status: 'denied',
|
|
111
|
+
message: data.message || data.error || `HTTP ${response.status}`,
|
|
112
|
+
};
|
|
113
|
+
}
|
|
114
|
+
if (!response.ok) {
|
|
115
|
+
return { status: 'error', message: data.message || data.error || `HTTP ${response.status}` };
|
|
116
|
+
}
|
|
117
|
+
if (!data.request_id) {
|
|
118
|
+
return { status: 'error', message: 'Trigger response did not include a request id' };
|
|
119
|
+
}
|
|
120
|
+
if (data.status === 'already_processing') {
|
|
121
|
+
return { status: 'already_processing', requestId: data.request_id };
|
|
122
|
+
}
|
|
123
|
+
return { status: 'started', requestId: data.request_id };
|
|
124
|
+
}
|
|
125
|
+
// Terminal states of the v3 status vocabulary (map_status_for_frontend in
|
|
126
|
+
// haystack-v3-request-status). Everything else — QUEUED, PROCESSING,
|
|
127
|
+
// CLONING, PREPARING, ANALYZING, not_found (row lag), and any state a newer
|
|
128
|
+
// backend introduces — reads as pending: misclassifying a healthy
|
|
129
|
+
// in-progress state as an error would abort the wait.
|
|
130
|
+
const TERMINAL_STATUSES = {
|
|
131
|
+
COMPLETED: 'COMPLETED',
|
|
132
|
+
FAILED: 'FAILED',
|
|
133
|
+
TIMED_OUT: 'FAILED',
|
|
134
|
+
SUPERSEDED: 'SUPERSEDED',
|
|
135
|
+
};
|
|
136
|
+
/**
|
|
137
|
+
* Poll the status of a triggered run by request id. The PR-level result
|
|
138
|
+
* endpoint can't distinguish a fresh re-run on the same head from the
|
|
139
|
+
* previous completed analysis — the request id can.
|
|
140
|
+
*/
|
|
141
|
+
export async function fetchTriggerStatus(requestId, token) {
|
|
142
|
+
const url = `${ANALYSIS_API_BASE}/api/analysis-trigger/status/${requestId}`;
|
|
143
|
+
try {
|
|
144
|
+
const response = await fetch(url, { headers: apiHeaders(token) });
|
|
145
|
+
// The status Lambda returns not_found as a 404 WITH a JSON body — parse
|
|
146
|
+
// before rejecting on status code so "not yet written" is distinguishable
|
|
147
|
+
// from a real transport failure.
|
|
148
|
+
let data = {};
|
|
149
|
+
try {
|
|
150
|
+
data = await response.json();
|
|
151
|
+
}
|
|
152
|
+
catch {
|
|
153
|
+
return { status: 'error', message: `HTTP ${response.status}` };
|
|
154
|
+
}
|
|
155
|
+
const status = typeof data.status === 'string' ? data.status : '';
|
|
156
|
+
const terminal = TERMINAL_STATUSES[status];
|
|
157
|
+
if (terminal) {
|
|
158
|
+
return { status: terminal, message: data.message };
|
|
159
|
+
}
|
|
160
|
+
// Only successful responses (plus the documented 404 not_found row-lag
|
|
161
|
+
// case) read as pending — a 5xx carrying {"status":"error"} must count
|
|
162
|
+
// toward the error budget, not spin the wait for the full timeout.
|
|
163
|
+
if (response.ok || (response.status === 404 && status === 'not_found')) {
|
|
164
|
+
return { status: 'pending', message: data.message };
|
|
165
|
+
}
|
|
166
|
+
return { status: 'error', message: data.message || `HTTP ${response.status}` };
|
|
167
|
+
}
|
|
168
|
+
catch (err) {
|
|
169
|
+
return { status: 'error', message: err.message };
|
|
170
|
+
}
|
|
171
|
+
}
|
|
67
172
|
/**
|
|
68
173
|
* Check if analysis results are available for a PR by polling the result endpoint.
|
|
69
174
|
* Returns 'ready' if complete, 'pending' if not yet available, or 'error' for server failures.
|