@memnexus-ai/mx-agent-cli 0.1.278 → 0.1.279
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -117,14 +117,104 @@ the only defense.
|
|
|
117
117
|
(#5086).
|
|
118
118
|
- A comment count or timeline thinner than you expect is a staleness signal, not a quiet PR.
|
|
119
119
|
- Name the head SHA you actually tested in the QA report: "PASS at 2dd3845d1", not "PASS on
|
|
120
|
-
#5046".
|
|
121
|
-
not
|
|
120
|
+
#5046". Fetch the PR head and capture the SHA in the same breath — on **every** test run,
|
|
121
|
+
not only after a force-push, because every verdict needs it:
|
|
122
|
+
|
|
123
|
+
```bash
|
|
124
|
+
# Pin the ref you test into a name of your own. Never carry a verdict on FETCH_HEAD.
|
|
125
|
+
git fetch origin --force "refs/pull/<N>/head:refs/pr/<N>" && \
|
|
126
|
+
HEAD_SHA=$(git rev-parse "refs/pr/<N>") # --force: this ref is not remote-tracking,
|
|
127
|
+
# so without it a re-review after a force-push
|
|
128
|
+
# is REJECTED and rev-parse returns the OLD head
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
**Never `git rev-parse FETCH_HEAD` for a verdict SHA.** `FETCH_HEAD` is one mutable file
|
|
132
|
+
per worktree, rewritten by the next fetch from any source, and it fails three ways that
|
|
133
|
+
are all routine rather than exotic:
|
|
134
|
+
- The `git fetch origin main` in the run-validity block below overwrites it. Do not
|
|
135
|
+
interleave these two procedures while holding an unpinned SHA.
|
|
136
|
+
- A bare `git fetch origin` with no refspec — what most agents run at session start —
|
|
137
|
+
writes every remote branch into it. Measured here: 158 entries, all `not-for-merge`,
|
|
138
|
+
and `git rev-parse FETCH_HEAD` returned line 155, an unrelated worktree branch. You
|
|
139
|
+
sit on your own branch and test other teams' branches, so this is the normal case.
|
|
140
|
+
- A second agent fetching in the same worktree reassigns it between your fetch and your
|
|
141
|
+
read. That happened during the review of this very file, with two reviewers active.
|
|
142
|
+
|
|
143
|
+
Do not take the SHA from `gh pr view --json headRefOid` either — that is the same API
|
|
144
|
+
surface that served a stale head on #5086. Every one of these failures yields a real,
|
|
145
|
+
head-shaped SHA, so the verdict still *looks* SHA-scoped. A SHA-scoped verdict stays
|
|
146
|
+
honest after the next force-push; a PR-scoped one does not, and a wrongly-captured one
|
|
147
|
+
only looks scoped.
|
|
122
148
|
- When checks were re-run to pick up a fix on main: a `pull_request` run executes
|
|
123
149
|
`refs/pull/N/merge`, not main, and reopening a PR does not reliably refresh that ref.
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
150
|
+
Establish which base the runner actually used by reading the job log, not the API summary:
|
|
151
|
+
|
|
152
|
+
```bash
|
|
153
|
+
# 0. This check applies to `pull_request` runs only — they are the ones that
|
|
154
|
+
# execute refs/pull/N/merge. Confirm the trigger before anything else.
|
|
155
|
+
gh run view <run_id> --json event -q .event # expect: pull_request
|
|
156
|
+
|
|
157
|
+
# 1. A run URL gives you a run ID; jobs have their own IDs. Require COMPLETED,
|
|
158
|
+
# and exclude jobs that never ran — a skipped job's log endpoint returns 404
|
|
159
|
+
# with an XML body that greps as silence.
|
|
160
|
+
gh api --paginate repos/OWNER/REPO/actions/runs/<run_id>/jobs \
|
|
161
|
+
--jq '.jobs[] | select(.status == "completed" and .conclusion != "skipped") | {id, name}'
|
|
162
|
+
# Choose the job by NAME: the one behind the check you actually care about.
|
|
163
|
+
# A matrix run has several and they are not guaranteed to share a checkout.
|
|
164
|
+
|
|
165
|
+
# 2. Assert the base THAT job checked out. This must SUCCEED — it is not a
|
|
166
|
+
# scan for a failure line. Run it against a job that actually checked out
|
|
167
|
+
# code: a `skipped` job and a `push`-triggered run both produce NO line, and
|
|
168
|
+
# no line is NOT a pass. It means you cannot judge this run from this job —
|
|
169
|
+
# pick another job, or another run.
|
|
170
|
+
BASE=$(gh api repos/OWNER/REPO/actions/jobs/<job_id>/logs \
|
|
171
|
+
| grep -ao 'Merge [0-9a-f]\{40\} into [0-9a-f]\{40\}' \
|
|
172
|
+
| sed 's/.* into //' | sort -u)
|
|
173
|
+
[ "$(printf '%s' "$BASE" | grep -c .)" -eq 1 ] \
|
|
174
|
+
|| { echo "STUCK: found $(printf '%s' "$BASE" | grep -c .) checkout lines, need exactly 1"; exit 1; }
|
|
175
|
+
|
|
176
|
+
# 3. Compare against a FRESHLY fetched main. Skip the fetch and you compare
|
|
177
|
+
# against a stale local ref, which can agree by accident.
|
|
178
|
+
git fetch origin main
|
|
179
|
+
[ "$BASE" = "$(git rev-parse origin/main)" ] || { echo "STALE BASE — run proved nothing"; exit 1; }
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
A stale base means the run proved nothing — close/reopen and re-check the next run's log.
|
|
183
|
+
"It ran after the fix landed" is not evidence it ran the fix (#5116).
|
|
184
|
+
- **No line found is not a pass.** State the check as an assertion that must succeed, never
|
|
185
|
+
as a scan for a failure line, because empty output is the *default* result in three
|
|
186
|
+
reproducible cases: a skipped job, a `push`-triggered run, and a 404 log body (stderr
|
|
187
|
+
only, empty stdout, exit 1). In all three a mismatch test never fires and the reviewer
|
|
188
|
+
sees nothing that fails. A `push` run checks out the branch tip directly and logs
|
|
189
|
+
`Switched to a new branch` — the merge-ref rule does not apply to it, and neither does
|
|
190
|
+
its reassurance. Match the anchored `Merge <40hex> into <40hex>` form rather than the
|
|
191
|
+
prose prefix: this document itself contains that prose, and an unanchored substring grep
|
|
192
|
+
can eventually match our own guidance instead of a runner's log.
|
|
193
|
+
- **Require the job to have COMPLETED, not merely to be un-skipped.** A running job has
|
|
194
|
+
`conclusion: null` and passes a not-skipped test, so the old filter returned it (measured
|
|
195
|
+
on job `92503438573`). It does not produce a wrong pass: `/actions/jobs/<id>/logs` serves
|
|
196
|
+
a COMPLETED log archive, never a partial one. Measured on the same job before and after
|
|
197
|
+
it finished — HTTP 404 `BlobNotFound` while running, 180 KB containing the base line once
|
|
198
|
+
complete — and confirmed not to be a checkout-timing artifact: a job 213 seconds in, with
|
|
199
|
+
`Checkout code` and seven later steps already `completed`, still returned the 404. So a
|
|
200
|
+
running job greps as silence and lands in the STUCK branch above.
|
|
201
|
+
Require `completed` anyway, because STUCK does not tell you WHICH problem you have. A
|
|
202
|
+
skipped job, a running job, and a purged log all return HTTP 404 with a near-identical
|
|
203
|
+
`BlobNotFound` body — and they call for three different actions (pick another job; wait;
|
|
204
|
+
treat the run as unauditable). Filtering on `completed` removes one of the three up front.
|
|
205
|
+
A job whose conclusion is unknown cannot tell you whether the run validated the fix,
|
|
206
|
+
whatever its log says. If the filter returns nothing, every job is still running — wait.
|
|
207
|
+
- **Retraction — do not use `merge_commit_sha`.** An earlier version of this rule said to
|
|
208
|
+
poll `gh api repos/OWNER/REPO/pulls/N --jq .merge_commit_sha` until it changed. That
|
|
209
|
+
check is wrong and is withdrawn. Measured on #4942: the API reported a *fresh*
|
|
210
|
+
`merge_commit_sha` while the runner still checked out a stale base — run `31063081630`,
|
|
211
|
+
base `3c226df8c`, 36 commits behind the good run's base `5f2bc52d3`, still carrying the
|
|
212
|
+
broken step body — and the run went green, 2.5 minutes and identical check names apart
|
|
213
|
+
from the real one. If you learned the polling rule, unlearn it.
|
|
214
|
+
- Nothing else about the run is load-bearing. On #4942 the check name, the conclusion, the
|
|
215
|
+
check rollup, the timestamp, and `merge_commit_sha` were all **identical** between the
|
|
216
|
+
worthless run and the real one. The job log's `into <base>` was the only field that
|
|
217
|
+
distinguished them.
|
|
128
218
|
|
|
129
219
|
### 3. Review Generated Tests
|
|
130
220
|
|
|
@@ -89,14 +89,121 @@ you read is the only defense.
|
|
|
89
89
|
(#5086).
|
|
90
90
|
- A comment count or timeline thinner than you expect is a staleness signal, not a quiet PR.
|
|
91
91
|
- Name the head SHA you actually read in the verdict: "PASS at 2dd3845d1", not "PASS on
|
|
92
|
-
#5046".
|
|
93
|
-
not
|
|
92
|
+
#5046". Fetch the PR head and capture the SHA in the same breath — on **every** review,
|
|
93
|
+
not only after a force-push, because every verdict needs it:
|
|
94
|
+
|
|
95
|
+
```bash
|
|
96
|
+
# Pin the ref you review into a name of your own. Never carry a verdict on FETCH_HEAD.
|
|
97
|
+
git fetch origin --force "refs/pull/<N>/head:refs/pr/<N>" && \
|
|
98
|
+
HEAD_SHA=$(git rev-parse "refs/pr/<N>") # --force: this ref is not remote-tracking,
|
|
99
|
+
# so without it a re-review after a force-push
|
|
100
|
+
# is REJECTED and rev-parse returns the OLD head
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
**Never `git rev-parse FETCH_HEAD` for a verdict SHA.** `FETCH_HEAD` is one mutable file
|
|
104
|
+
per worktree, rewritten by the next fetch from any source, and it fails three ways that
|
|
105
|
+
are all routine rather than exotic:
|
|
106
|
+
- The `git fetch origin main` in the run-validity block below overwrites it. Do not
|
|
107
|
+
interleave these two procedures while holding an unpinned SHA.
|
|
108
|
+
- A bare `git fetch origin` with no refspec — what most agents run at session start —
|
|
109
|
+
writes every remote branch into it. Measured here: 158 entries, all `not-for-merge`,
|
|
110
|
+
and `git rev-parse FETCH_HEAD` returned line 155, an unrelated worktree branch. You
|
|
111
|
+
sit on your own branch and review other teams' branches, so this is the normal case.
|
|
112
|
+
- A second agent fetching in the same worktree reassigns it between your fetch and your
|
|
113
|
+
read. That happened during the review of this very file, with two reviewers active.
|
|
114
|
+
|
|
115
|
+
Do not take the SHA from `gh pr view --json headRefOid` either — that is the same API
|
|
116
|
+
surface that served a stale head on #5086. Every one of these failures yields a real,
|
|
117
|
+
head-shaped SHA, so the verdict still *looks* SHA-scoped. A SHA-scoped verdict stays
|
|
118
|
+
honest after the next force-push; a PR-scoped one does not, and a wrongly-captured one
|
|
119
|
+
only looks scoped.
|
|
94
120
|
- When checks were re-run to pick up a fix on main: a `pull_request` run executes
|
|
95
121
|
`refs/pull/N/merge`, not main, and reopening a PR does not reliably refresh that ref.
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
122
|
+
Establish which base the runner actually used by reading the job log, not the API summary:
|
|
123
|
+
|
|
124
|
+
```bash
|
|
125
|
+
# 0. This check applies to `pull_request` runs only — they are the ones that
|
|
126
|
+
# execute refs/pull/N/merge. Confirm the trigger before anything else.
|
|
127
|
+
gh run view <run_id> --json event -q .event # expect: pull_request
|
|
128
|
+
|
|
129
|
+
# 1. A run URL gives you a run ID; jobs have their own IDs. Require COMPLETED,
|
|
130
|
+
# and exclude jobs that never ran — a skipped job's log endpoint returns 404
|
|
131
|
+
# with an XML body that greps as silence.
|
|
132
|
+
gh api --paginate repos/OWNER/REPO/actions/runs/<run_id>/jobs \
|
|
133
|
+
--jq '.jobs[] | select(.status == "completed" and .conclusion != "skipped") | {id, name}'
|
|
134
|
+
# Choose the job by NAME: the one behind the check you actually care about.
|
|
135
|
+
# A matrix run has several and they are not guaranteed to share a checkout.
|
|
136
|
+
|
|
137
|
+
# 2. Assert the base THAT job checked out. This must SUCCEED — it is not a
|
|
138
|
+
# scan for a failure line. Run it against a job that actually checked out
|
|
139
|
+
# code: a `skipped` job and a `push`-triggered run both produce NO line, and
|
|
140
|
+
# no line is NOT a pass. It means you cannot judge this run from this job —
|
|
141
|
+
# pick another job, or another run.
|
|
142
|
+
BASE=$(gh api repos/OWNER/REPO/actions/jobs/<job_id>/logs \
|
|
143
|
+
| grep -ao 'Merge [0-9a-f]\{40\} into [0-9a-f]\{40\}' \
|
|
144
|
+
| sed 's/.* into //' | sort -u)
|
|
145
|
+
[ "$(printf '%s' "$BASE" | grep -c .)" -eq 1 ] \
|
|
146
|
+
|| { echo "STUCK: found $(printf '%s' "$BASE" | grep -c .) checkout lines, need exactly 1"; exit 1; }
|
|
147
|
+
|
|
148
|
+
# 3. Compare against a FRESHLY fetched main. Skip the fetch and you compare
|
|
149
|
+
# against a stale local ref, which can agree by accident.
|
|
150
|
+
git fetch origin main
|
|
151
|
+
[ "$BASE" = "$(git rev-parse origin/main)" ] || { echo "STALE BASE — run proved nothing"; exit 1; }
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
A stale base means the run proved nothing — close/reopen and re-check the next run's log.
|
|
155
|
+
"It ran after the fix landed" is not evidence it ran the fix (#5116).
|
|
156
|
+
- **No line found is not a pass.** State the check as an assertion that must succeed, never
|
|
157
|
+
as a scan for a failure line, because empty output is the *default* result in three
|
|
158
|
+
reproducible cases: a skipped job, a `push`-triggered run, and a 404 log body (stderr
|
|
159
|
+
only, empty stdout, exit 1). In all three a mismatch test never fires and the reviewer
|
|
160
|
+
sees nothing that fails. A `push` run checks out the branch tip directly and logs
|
|
161
|
+
`Switched to a new branch` — the merge-ref rule does not apply to it, and neither does
|
|
162
|
+
its reassurance. Match the anchored `Merge <40hex> into <40hex>` form rather than the
|
|
163
|
+
prose prefix: this document itself contains that prose, and an unanchored substring grep
|
|
164
|
+
can eventually match our own guidance instead of a runner's log.
|
|
165
|
+
- **Require the job to have COMPLETED, not merely to be un-skipped.** A running job has
|
|
166
|
+
`conclusion: null` and passes a not-skipped test, so the old filter returned it (measured
|
|
167
|
+
on job `92503438573`). It does not produce a wrong pass: `/actions/jobs/<id>/logs` serves
|
|
168
|
+
a COMPLETED log archive, never a partial one. Measured on the same job before and after
|
|
169
|
+
it finished — HTTP 404 `BlobNotFound` while running, 180 KB containing the base line once
|
|
170
|
+
complete — and confirmed not to be a checkout-timing artifact: a job 213 seconds in, with
|
|
171
|
+
`Checkout code` and seven later steps already `completed`, still returned the 404. So a
|
|
172
|
+
running job greps as silence and lands in the STUCK branch above.
|
|
173
|
+
Require `completed` anyway, because STUCK does not tell you WHICH problem you have. A
|
|
174
|
+
skipped job, a running job, and a purged log all return HTTP 404 with a near-identical
|
|
175
|
+
`BlobNotFound` body — and they call for three different actions (pick another job; wait;
|
|
176
|
+
treat the run as unauditable). Filtering on `completed` removes one of the three up front.
|
|
177
|
+
A job whose conclusion is unknown cannot tell you whether the run validated the fix,
|
|
178
|
+
whatever its log says. If the filter returns nothing, every job is still running — wait.
|
|
179
|
+
- **Retraction — do not use `merge_commit_sha`.** An earlier version of this rule said to
|
|
180
|
+
poll `gh api repos/OWNER/REPO/pulls/N --jq .merge_commit_sha` until it changed. That
|
|
181
|
+
check is wrong and is withdrawn. Measured on #4942: the API reported a *fresh*
|
|
182
|
+
`merge_commit_sha` while the runner still checked out a stale base — run `31063081630`,
|
|
183
|
+
base `3c226df8c`, 36 commits behind the good run's base `5f2bc52d3`, still carrying the
|
|
184
|
+
broken step body — and the run went green, 2.5 minutes and identical check names apart
|
|
185
|
+
from the real one. If you learned the polling rule, unlearn it.
|
|
186
|
+
- Nothing else about the run is load-bearing. On #4942 the check name, the conclusion, the
|
|
187
|
+
check rollup, the timestamp, and `merge_commit_sha` were all **identical** between the
|
|
188
|
+
worthless run and the real one. The job log's `into <base>` was the only field that
|
|
189
|
+
distinguished them.
|
|
190
|
+
- The Codex differential review is a locally-run script, not a CI job, so "a comment
|
|
191
|
+
exists" is not evidence it ran clean. Check the **exit status**, and record the SHA
|
|
192
|
+
yourself:
|
|
193
|
+
|
|
194
|
+
```bash
|
|
195
|
+
dev-tools/scripts/codex-review.sh --base origin/<target-branch> out.txt; echo "rc=$?"
|
|
196
|
+
git rev-parse HEAD # record alongside — the artifact will not give you this
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
Non-zero is the whole signal. `dev-tools/scripts/codex-review.sh:715-717` exits
|
|
200
|
+
`OUTPUT_GUARD_EXIT` when the output guard withheld or redacted the review, and `codex_rc`
|
|
201
|
+
otherwise; only `rc=0` is a clean review. **The reviewed SHA is not obtainable from the
|
|
202
|
+
artifact in the mode you are told to use.** The `/codex` skill and the script both push
|
|
203
|
+
you to `--base` (`--commit` is refused unless HEAD carries a single commit or you pass
|
|
204
|
+
`--partial-ok`), and in `--base` mode the `REVIEW SCOPE` header names a *branch* and a
|
|
205
|
+
commit count, no SHA at either end. So the SHA half of this check is unenforceable from
|
|
206
|
+
the artifact alone — you must record `git rev-parse HEAD` next to the run, or it is lost.
|
|
100
207
|
|
|
101
208
|
For each changed file, apply the appropriate security lens:
|
|
102
209
|
|