@openai/codex-security 0.1.9 → 0.1.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +16 -5
- package/_bundled_plugin/.codex-plugin/plugin.json +1 -1
- package/_bundled_plugin/mcp/server.mjs.br.part-000 +0 -0
- package/_bundled_plugin/mcp/server.mjs.br.part-001 +0 -0
- package/_bundled_plugin/preflight/capability-profiles.toml +1 -1
- package/_bundled_plugin/references/scan-contract.md +1 -1
- package/_bundled_plugin/scripts/deep_scan_workbench.py +22 -0
- package/_bundled_plugin/scripts/generate_in_scope_files.py +102 -1
- package/_bundled_plugin/scripts/normalize_candidates.py +20 -2
- package/_bundled_plugin/scripts/workbench_cli.py +4 -0
- package/_bundled_plugin/scripts/workbench_db.py +12 -0
- package/_bundled_plugin/scripts/workbench_native_indexes.py +100 -43
- package/_bundled_plugin/skills/finding-discovery/SKILL.md +8 -4
- package/_bundled_plugin/skills/security-diff-scan/SKILL.md +21 -140
- package/dist/api.d.ts +42 -3
- package/dist/api.d.ts.map +1 -1
- package/dist/api.js +103 -44
- package/dist/api.js.map +1 -1
- package/dist/cli.d.ts +1 -1
- package/dist/cli.d.ts.map +1 -1
- package/dist/cli.js +93 -39
- package/dist/cli.js.map +1 -1
- package/dist/cost.d.ts +1 -0
- package/dist/cost.d.ts.map +1 -1
- package/dist/cost.js +1 -1
- package/dist/cost.js.map +1 -1
- package/dist/errors.d.ts +4 -2
- package/dist/errors.d.ts.map +1 -1
- package/dist/errors.js +10 -46
- package/dist/errors.js.map +1 -1
- package/dist/index.d.ts +2 -2
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -1
- package/dist/index.js.map +1 -1
- package/dist/multiscan.js +2 -2
- package/dist/multiscan.js.map +1 -1
- package/dist/result.d.ts +13 -1
- package/dist/result.d.ts.map +1 -1
- package/dist/result.js +3 -0
- package/dist/result.js.map +1 -1
- package/dist/runtime.js +3 -3
- package/dist/runtime.js.map +1 -1
- package/dist/scan-comparison.d.ts +14 -0
- package/dist/scan-comparison.d.ts.map +1 -1
- package/dist/scan-comparison.js +89 -0
- package/dist/scan-comparison.js.map +1 -1
- package/dist/scan-dashboard.d.ts.map +1 -1
- package/dist/scan-dashboard.js +1 -0
- package/dist/scan-dashboard.js.map +1 -1
- package/dist/scan-history-renderer.d.ts +1 -1
- package/dist/scan-history-renderer.d.ts.map +1 -1
- package/dist/scan-history-renderer.js +10 -0
- package/dist/scan-history-renderer.js.map +1 -1
- package/dist/scan-logs.d.ts +17 -0
- package/dist/scan-logs.d.ts.map +1 -0
- package/dist/scan-logs.js +90 -0
- package/dist/scan-logs.js.map +1 -0
- package/dist/version.d.ts +1 -1
- package/dist/version.js +1 -1
- package/dist/version.js.map +1 -1
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -53,6 +53,10 @@ Use `security.preflight()` to validate local inputs, `onWorkerStatus` and
|
|
|
53
53
|
`onReconnect` to observe long-running scans, and an `AbortSignal` to cancel a
|
|
54
54
|
scan.
|
|
55
55
|
|
|
56
|
+
Successful results include open repository findings in `repositoryFindings`,
|
|
57
|
+
when available; `findings` remains the current scan. Matching earlier findings
|
|
58
|
+
can make one additional model call, including with a scan cost limit.
|
|
59
|
+
|
|
56
60
|
Results can contain source excerpts, vulnerability details, and reproduction
|
|
57
61
|
steps. Keep result directories and saved reports outside the repository and
|
|
58
62
|
limit access to authorized reviewers.
|
|
@@ -218,10 +222,12 @@ npx @openai/codex-security bulk-scan repositories.csv --output-dir /path/outside
|
|
|
218
222
|
npx @openai/codex-security scans list /path/to/repository
|
|
219
223
|
npx @openai/codex-security scans list --scan-root /path/outside/repository/results
|
|
220
224
|
npx @openai/codex-security scans show SCAN_ID
|
|
225
|
+
npx @openai/codex-security scans logs SCAN_ID
|
|
221
226
|
npx @openai/codex-security scans rerun SCAN_ID
|
|
222
227
|
npx @openai/codex-security scans match PREVIOUS_SCAN_ID CURRENT_SCAN_ID
|
|
223
228
|
npx @openai/codex-security scans match --all
|
|
224
229
|
npx @openai/codex-security scans compare PREVIOUS_SCAN_ID CURRENT_SCAN_ID
|
|
230
|
+
npx @openai/codex-security findings list /path/to/repository
|
|
225
231
|
npx @openai/codex-security findings false-positive OCCURRENCE_ID --reason "The route already checks permissions"
|
|
226
232
|
npx @openai/codex-security export /path/outside/repository/results --export-format sarif --output /path/outside/repository/results.sarif
|
|
227
233
|
npx @openai/codex-security export /path/outside/repository/results --export-format csv --output /path/outside/repository/findings.csv
|
|
@@ -400,7 +406,7 @@ The CLI and SDK recognize the following user-configurable environment:
|
|
|
400
406
|
| Variable | Effect |
|
|
401
407
|
| --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
|
|
402
408
|
| `OPENAI_API_KEY`, `CODEX_API_KEY` | Scan authentication; `OPENAI_API_KEY` wins when both are present. |
|
|
403
|
-
| `CODEX_SECURITY_LOG_LEVEL` | CLI-only; set to `debug` for
|
|
409
|
+
| `CODEX_SECURITY_LOG_LEVEL` | CLI-only; set to `debug` for verbose diagnostics. |
|
|
404
410
|
| `LOG_LEVEL` | CLI-only fallback when `CODEX_SECURITY_LOG_LEVEL` is unset. |
|
|
405
411
|
| `CODEX_SECURITY_STATE_DIR` | Override the private scan-history, workbench, and default artifact directory. |
|
|
406
412
|
| `CODEX_HOME` | Set the ambient Codex home for file-backed sign-in and default state; defaults to `~/.codex`. |
|
|
@@ -451,11 +457,12 @@ token and worker counts, estimated cost, the results directory, and the next
|
|
|
451
457
|
useful command.
|
|
452
458
|
Progress and summaries use stderr; structured scan results remain on stdout.
|
|
453
459
|
|
|
454
|
-
Add `--verbose` or set `CODEX_SECURITY_LOG_LEVEL=debug` to print
|
|
460
|
+
Add `--verbose` or set `CODEX_SECURITY_LOG_LEVEL=debug` to print
|
|
455
461
|
lifecycle, authentication, progress, and cost diagnostics to stderr.
|
|
456
462
|
`LOG_LEVEL=debug` is used only when `CODEX_SECURITY_LOG_LEVEL` is unset.
|
|
457
|
-
|
|
458
|
-
|
|
463
|
+
Structured JSON results remain on stdout. Verbose diagnostics may contain
|
|
464
|
+
sensitive data; review local logs before sharing them. The interactive
|
|
465
|
+
dashboard omits activity containing recognizable credentials.
|
|
459
466
|
|
|
460
467
|
Each scan records its model, tokens, and estimated cost in its JSON result,
|
|
461
468
|
scan history, and bulk-scan receipt. Estimates use
|
|
@@ -508,13 +515,17 @@ whose artifacts are under a particular root. `scans show SCAN_ID` includes the
|
|
|
508
515
|
scan configuration, results, coverage, and artifact locations. Add
|
|
509
516
|
`--show-linked-findings` to include finding links from previous scans.
|
|
510
517
|
|
|
518
|
+
`scans logs SCAN_ID` shows complete session events from the scan and its
|
|
519
|
+
workers, which can include source code and credentials.
|
|
520
|
+
|
|
511
521
|
Every scan history command accepts a full scan ID or a unique prefix of at
|
|
512
522
|
least eight characters.
|
|
513
523
|
|
|
514
524
|
Scan history uses the existing Codex Security workbench database at
|
|
515
525
|
`$CODEX_HOME/state/plugins/codex-security/workbench.sqlite3`. Set
|
|
516
526
|
`CODEX_SECURITY_STATE_DIR` to place the database elsewhere. Scan credentials
|
|
517
|
-
are never stored in the scan configuration.
|
|
527
|
+
are never stored in the scan configuration. Recorded failure summaries and
|
|
528
|
+
bulk-scan receipts omit messages that contain recognizable credentials.
|
|
518
529
|
|
|
519
530
|
The scan sandbox permits writes to the selected state directory so SQLite can
|
|
520
531
|
maintain its database and journal files. If the host itself cannot write to the
|
|
Binary file
|
|
Binary file
|
|
@@ -40,7 +40,7 @@ description = "Capabilities for Git-backed Codex Security diff scans."
|
|
|
40
40
|
[[profiles.security_diff_scan.requirements]]
|
|
41
41
|
capability = "delegated_workers"
|
|
42
42
|
severity = "warn"
|
|
43
|
-
reason = "Large
|
|
43
|
+
reason = "Large diff scans may use discovery workers when available; otherwise the parent reviews every changed file."
|
|
44
44
|
|
|
45
45
|
[[profiles.security_diff_scan.requirements]]
|
|
46
46
|
capability = "goal_tools"
|
|
@@ -44,7 +44,7 @@ A dirty checkout has `allowedKinds: ["git_worktree"]`: copy `requiredSnapshotDig
|
|
|
44
44
|
|
|
45
45
|
`targetId` identifies the stable repository or workspace. Prefer a digest of a sanitized canonical absolute remote URL when one exists. Otherwise use a digest of a stable local workspace identity. Never persist remote URL credentials, query parameters, fragments, or tokens.
|
|
46
46
|
|
|
47
|
-
For dirty worktrees and diffs, calculate `snapshotDigest` from a deterministic representation of the reviewed content, including staged changes and reviewed untracked files where applicable. For directory snapshots, hash a sorted relative-path and file-hash inventory of the reviewed scope. Encode the result as `codex-security-snapshot/v1:sha256:<64 lowercase hex characters>`.
|
|
47
|
+
For dirty worktrees and working-tree diffs, calculate `snapshotDigest` from a deterministic representation of the reviewed content, including staged changes and reviewed untracked files where applicable. For committed or revision-range diffs, derive it from the exact authoritative diff kind and immutable base/head revisions. For directory snapshots, hash a sorted relative-path and file-hash inventory of the reviewed scope. Encode the result as `codex-security-snapshot/v1:sha256:<64 lowercase hex characters>`.
|
|
48
48
|
|
|
49
49
|
## Finding Identity
|
|
50
50
|
|
|
@@ -1274,6 +1274,28 @@ def upsert_deep_scan_worker(
|
|
|
1274
1274
|
""",
|
|
1275
1275
|
(timestamp, scan_id),
|
|
1276
1276
|
)
|
|
1277
|
+
elif args.kind == "dedup" and args.status == "failed" and existing["status"] == "running":
|
|
1278
|
+
connection.execute(
|
|
1279
|
+
"""
|
|
1280
|
+
UPDATE deep_scan_workers
|
|
1281
|
+
SET merge_state = 'buffered', updated_at = ?
|
|
1282
|
+
WHERE scan_id = ? AND kind = 'discovery' AND status = 'succeeded'
|
|
1283
|
+
AND merge_state = 'merging'
|
|
1284
|
+
AND id IN (
|
|
1285
|
+
SELECT discovery_worker_id FROM deep_scan_dedup_inputs
|
|
1286
|
+
WHERE scan_id = ? AND dedup_worker_id = ?
|
|
1287
|
+
)
|
|
1288
|
+
""",
|
|
1289
|
+
(timestamp, scan_id, scan_id, worker_id),
|
|
1290
|
+
)
|
|
1291
|
+
connection.execute(
|
|
1292
|
+
"""
|
|
1293
|
+
UPDATE deep_scan_runs
|
|
1294
|
+
SET phase = 'discovery', updated_at = ?
|
|
1295
|
+
WHERE scan_id = ?
|
|
1296
|
+
""",
|
|
1297
|
+
(timestamp, scan_id),
|
|
1298
|
+
)
|
|
1277
1299
|
completed_at = (
|
|
1278
1300
|
timestamp
|
|
1279
1301
|
if args.status in {"succeeded", "failed", "canceled"}
|
|
@@ -91,6 +91,88 @@ def generate_in_scope_files(repository: Path, scope: str, output: Path) -> int:
|
|
|
91
91
|
inventory.seek(0)
|
|
92
92
|
rows = sorted(inventory)
|
|
93
93
|
|
|
94
|
+
return write_inventory(output, rows)
|
|
95
|
+
|
|
96
|
+
|
|
97
|
+
def generate_diff_in_scope_files(
|
|
98
|
+
repository: Path,
|
|
99
|
+
base: str,
|
|
100
|
+
head: str,
|
|
101
|
+
mode: str,
|
|
102
|
+
output: Path,
|
|
103
|
+
) -> int:
|
|
104
|
+
"""Reuse the existing diff selection without generating previews or duplicate worklists."""
|
|
105
|
+
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
|
106
|
+
from generate_rank_input import git_changed_paths, path_is_excluded, run_git_changed_paths
|
|
107
|
+
from rank_preview import (
|
|
108
|
+
DEFAULT_PREVIEW_BYTES,
|
|
109
|
+
TEXT_CODE_EXTENSIONS,
|
|
110
|
+
is_binary_sample,
|
|
111
|
+
preview_for,
|
|
112
|
+
)
|
|
113
|
+
|
|
114
|
+
rows: list[bytes] = []
|
|
115
|
+
try:
|
|
116
|
+
if mode == "local-patch":
|
|
117
|
+
changed = run_git_changed_paths(repository, [base])
|
|
118
|
+
untracked = subprocess.run(
|
|
119
|
+
["git", "-C", str(repository), "ls-files", "--others", "--exclude-standard", "-z"],
|
|
120
|
+
capture_output=True,
|
|
121
|
+
text=True,
|
|
122
|
+
check=True,
|
|
123
|
+
)
|
|
124
|
+
changed.extend(
|
|
125
|
+
(repository / relative, "A")
|
|
126
|
+
for relative in untracked.stdout.split("\0")
|
|
127
|
+
if relative
|
|
128
|
+
)
|
|
129
|
+
else:
|
|
130
|
+
changed = git_changed_paths(repository, base, head, mode)
|
|
131
|
+
|
|
132
|
+
for path, status in changed:
|
|
133
|
+
relative = path.relative_to(repository)
|
|
134
|
+
if path_is_excluded(relative) or path.suffix.lower() not in TEXT_CODE_EXTENSIONS:
|
|
135
|
+
continue
|
|
136
|
+
if status != "D":
|
|
137
|
+
if mode == "revisions":
|
|
138
|
+
contents = subprocess.run(
|
|
139
|
+
[
|
|
140
|
+
"git",
|
|
141
|
+
"-C",
|
|
142
|
+
str(repository),
|
|
143
|
+
"cat-file",
|
|
144
|
+
"blob",
|
|
145
|
+
f"{head}:{relative.as_posix()}",
|
|
146
|
+
],
|
|
147
|
+
capture_output=True,
|
|
148
|
+
check=True,
|
|
149
|
+
).stdout
|
|
150
|
+
if is_binary_sample(contents):
|
|
151
|
+
continue
|
|
152
|
+
elif (
|
|
153
|
+
path.is_symlink()
|
|
154
|
+
or not path.is_file()
|
|
155
|
+
or preview_for(path, DEFAULT_PREVIEW_BYTES)[1]
|
|
156
|
+
):
|
|
157
|
+
continue
|
|
158
|
+
relative_path = relative.as_posix()
|
|
159
|
+
if "\n" in relative_path or "\r" in relative_path:
|
|
160
|
+
raise InventoryError(
|
|
161
|
+
"Git changes contain a path that cannot fit in the file inventory"
|
|
162
|
+
)
|
|
163
|
+
rows.append(f"{relative_path}\n".encode())
|
|
164
|
+
except (OSError, subprocess.CalledProcessError) as error:
|
|
165
|
+
detail = getattr(error, "stderr", None)
|
|
166
|
+
if isinstance(detail, bytes):
|
|
167
|
+
detail = detail.decode("utf-8", errors="replace")
|
|
168
|
+
message = detail.strip() if isinstance(detail, str) and detail.strip() else str(error)
|
|
169
|
+
raise InventoryError(f"could not resolve the selected Git changes: {message}") from error
|
|
170
|
+
|
|
171
|
+
return write_inventory(output, sorted(set(rows)))
|
|
172
|
+
|
|
173
|
+
|
|
174
|
+
def write_inventory(output: Path, rows: list[bytes]) -> int:
|
|
175
|
+
"""Replace a complete inventory atomically, keeping failures from corrupting the old one."""
|
|
94
176
|
output.parent.mkdir(parents=True, exist_ok=True)
|
|
95
177
|
temporary: Path | None = None
|
|
96
178
|
try:
|
|
@@ -116,13 +198,32 @@ def main() -> None:
|
|
|
116
198
|
parser.add_argument("--repo", required=True, help="Repository root.")
|
|
117
199
|
parser.add_argument("--scope", required=True, help="File or directory within the repository.")
|
|
118
200
|
parser.add_argument("--out", required=True, help="Destination for the file inventory.")
|
|
201
|
+
parser.add_argument("--diff-base", help="Authoritative Git base for a changed-file inventory.")
|
|
202
|
+
parser.add_argument("--diff-head", default="HEAD", help="Authoritative Git head revision.")
|
|
203
|
+
parser.add_argument(
|
|
204
|
+
"--diff-mode",
|
|
205
|
+
choices=("revisions", "local-patch"),
|
|
206
|
+
default="revisions",
|
|
207
|
+
help="Use committed revisions or the current staged and unstaged patch.",
|
|
208
|
+
)
|
|
119
209
|
args = parser.parse_args()
|
|
120
210
|
|
|
121
211
|
try:
|
|
122
212
|
repository = resolve_repository(args.repo)
|
|
123
213
|
scope = resolve_scope(repository, args.scope)
|
|
124
214
|
output = resolve_output(args.out)
|
|
125
|
-
|
|
215
|
+
if args.diff_base is None:
|
|
216
|
+
count = generate_in_scope_files(repository, scope, output)
|
|
217
|
+
elif scope not in (".", "./"):
|
|
218
|
+
raise InventoryError("--scope: diff scans must use the repository root")
|
|
219
|
+
else:
|
|
220
|
+
count = generate_diff_in_scope_files(
|
|
221
|
+
repository,
|
|
222
|
+
args.diff_base,
|
|
223
|
+
args.diff_head,
|
|
224
|
+
args.diff_mode,
|
|
225
|
+
output,
|
|
226
|
+
)
|
|
126
227
|
except (OSError, ValueError) as error:
|
|
127
228
|
print(f"generate_in_scope_files: {error}", file=sys.stderr)
|
|
128
229
|
raise SystemExit(2) from error
|
|
@@ -135,7 +135,7 @@ def normalize_locations(
|
|
|
135
135
|
)
|
|
136
136
|
|
|
137
137
|
|
|
138
|
-
def read_scope(path: Path, repo_root: Path) -> set[str]:
|
|
138
|
+
def read_scope(path: Path, repo_root: Path, *, allow_missing: bool = False) -> set[str]:
|
|
139
139
|
contents = path.read_bytes().decode("utf-8")
|
|
140
140
|
lines = contents.split("\n")
|
|
141
141
|
listed_rows = set(lines)
|
|
@@ -183,6 +183,19 @@ def read_scope(path: Path, repo_root: Path) -> set[str]:
|
|
|
183
183
|
try:
|
|
184
184
|
relative, _ = relative_file(line, repo_root)
|
|
185
185
|
except (OSError, ValueError) as error:
|
|
186
|
+
if allow_missing and isinstance(error, FileNotFoundError):
|
|
187
|
+
candidate = PurePosixPath(line)
|
|
188
|
+
if candidate.is_absolute() or ".." in candidate.parts or "\0" in line:
|
|
189
|
+
raise ValueError(f"in-scope file row {number}: unsafe deleted path") from error
|
|
190
|
+
resolved = (repo_root / line).resolve(strict=False)
|
|
191
|
+
try:
|
|
192
|
+
relative = resolved.relative_to(repo_root).as_posix()
|
|
193
|
+
except ValueError as escaped:
|
|
194
|
+
raise ValueError(
|
|
195
|
+
f"in-scope file row {number}: path escapes repository"
|
|
196
|
+
) from escaped
|
|
197
|
+
scope.add(relative)
|
|
198
|
+
continue
|
|
186
199
|
raise ValueError(f"in-scope file row {number}: {error}") from error
|
|
187
200
|
scope.add(relative)
|
|
188
201
|
return scope
|
|
@@ -261,6 +274,11 @@ def main() -> None:
|
|
|
261
274
|
parser.add_argument("--out", required=True, help="Combined candidate JSONL output.")
|
|
262
275
|
parser.add_argument("--repo-root", required=True, help="Repository root for candidate paths.")
|
|
263
276
|
parser.add_argument("--in-scope-files", required=True, help="Repository-relative file list.")
|
|
277
|
+
parser.add_argument(
|
|
278
|
+
"--allow-missing-in-scope",
|
|
279
|
+
action="store_true",
|
|
280
|
+
help="Keep deleted Git paths in a diff inventory while validating existing candidate files.",
|
|
281
|
+
)
|
|
264
282
|
args = parser.parse_args()
|
|
265
283
|
try:
|
|
266
284
|
repo_root = Path(args.repo_root).expanduser().resolve(strict=True)
|
|
@@ -273,7 +291,7 @@ def main() -> None:
|
|
|
273
291
|
raise ValueError("--out: must not also be an input")
|
|
274
292
|
if output == scope_path:
|
|
275
293
|
raise ValueError("--out: must not replace --in-scope-files")
|
|
276
|
-
scope = read_scope(scope_path, repo_root)
|
|
294
|
+
scope = read_scope(scope_path, repo_root, allow_missing=args.allow_missing_in_scope)
|
|
277
295
|
line_counts: dict[Path, int] = {}
|
|
278
296
|
rows: list[dict[str, Any]] = []
|
|
279
297
|
for source in inputs:
|
|
@@ -147,6 +147,10 @@ def parse_args(description: str) -> argparse.Namespace:
|
|
|
147
147
|
register_cli_scan.add_argument("--archive-existing", action="store_true")
|
|
148
148
|
register_cli_scan.add_argument("--archived-scan-dir")
|
|
149
149
|
|
|
150
|
+
set_scan_thread = subparsers.add_parser("set-scan-thread")
|
|
151
|
+
set_scan_thread.add_argument("--scan-id", required=True)
|
|
152
|
+
set_scan_thread.add_argument("--thread-id", required=True)
|
|
153
|
+
|
|
150
154
|
get_scan_recipe = subparsers.add_parser("get-scan-recipe")
|
|
151
155
|
get_scan_recipe.add_argument("--scan-id", required=True)
|
|
152
156
|
|
|
@@ -1432,6 +1432,16 @@ def register_cli_scan(connection: sqlite3.Connection, args: argparse.Namespace)
|
|
|
1432
1432
|
}
|
|
1433
1433
|
|
|
1434
1434
|
|
|
1435
|
+
def set_scan_thread(connection: sqlite3.Connection, args: argparse.Namespace) -> dict[str, Any]:
|
|
1436
|
+
scan = require_scan(connection, args.scan_id)
|
|
1437
|
+
with connection:
|
|
1438
|
+
connection.execute(
|
|
1439
|
+
"UPDATE scans SET continuation_thread_id = ?, updated_at = ? WHERE id = ?",
|
|
1440
|
+
(args.thread_id, now(), scan["id"]),
|
|
1441
|
+
)
|
|
1442
|
+
return {"scanId": scan["id"], "threadId": args.thread_id}
|
|
1443
|
+
|
|
1444
|
+
|
|
1435
1445
|
def parse_scan_recipe(value: str, repository: Path) -> dict[str, Any]:
|
|
1436
1446
|
if len(value.encode("utf-8")) > SCAN_RECIPE_MAX_BYTES:
|
|
1437
1447
|
raise SystemExit("Scan launch recipe must be no larger than 256 KiB.")
|
|
@@ -3354,6 +3364,8 @@ def main() -> None:
|
|
|
3354
3364
|
)
|
|
3355
3365
|
elif args.command == "register-cli-scan":
|
|
3356
3366
|
result = register_cli_scan(connection, args)
|
|
3367
|
+
elif args.command == "set-scan-thread":
|
|
3368
|
+
result = set_scan_thread(connection, args)
|
|
3357
3369
|
elif args.command == "get-scan-recipe":
|
|
3358
3370
|
result = get_scan_recipe(connection, args)
|
|
3359
3371
|
elif args.command == "compare-scans":
|
|
@@ -47,9 +47,13 @@ def list_global_findings(
|
|
|
47
47
|
return {
|
|
48
48
|
"findings": [
|
|
49
49
|
{
|
|
50
|
+
"confirmedInLatestScan": row["confirmed_in_latest_scan"],
|
|
50
51
|
"createdAt": row["created_at"],
|
|
51
52
|
"findingId": row["finding_id"],
|
|
53
|
+
"knownSince": row["known_since"],
|
|
54
|
+
"knownScanIds": row["known_scan_ids"],
|
|
52
55
|
"locationPath": row["location_path"],
|
|
56
|
+
"matchedFindingIds": row["matched_finding_ids"],
|
|
53
57
|
"occurrenceCount": row["occurrence_count"],
|
|
54
58
|
"occurrenceId": row["occurrence_id"],
|
|
55
59
|
"scanId": row["scan_id"],
|
|
@@ -70,64 +74,117 @@ def list_global_findings(
|
|
|
70
74
|
}
|
|
71
75
|
|
|
72
76
|
|
|
73
|
-
def _indexed_findings(connection: sqlite3.Connection) -> Iterator[
|
|
74
|
-
|
|
77
|
+
def _indexed_findings(connection: sqlite3.Connection) -> Iterator[dict[str, Any]]:
|
|
78
|
+
parents: dict[tuple[str, str], tuple[str, str]] = {}
|
|
79
|
+
|
|
80
|
+
def group(identity: tuple[str, str]) -> tuple[str, str]:
|
|
81
|
+
while identity in parents:
|
|
82
|
+
identity = parents[identity]
|
|
83
|
+
return identity
|
|
84
|
+
|
|
85
|
+
for match in connection.execute(
|
|
75
86
|
"""
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
FROM finding_occurrences AS occurrences
|
|
96
|
-
JOIN scans ON scans.id = occurrences.scan_id
|
|
97
|
-
JOIN security_targets AS targets ON targets.id = scans.target_id
|
|
98
|
-
LEFT JOIN finding_triage AS triage ON triage.occurrence_id = occurrences.id
|
|
87
|
+
SELECT before_scans.target_id, before.finding_id AS before_finding_id,
|
|
88
|
+
after.finding_id AS after_finding_id
|
|
89
|
+
FROM scan_comparison_matches AS matches
|
|
90
|
+
JOIN finding_occurrences AS before ON before.id = matches.before_occurrence_id
|
|
91
|
+
JOIN scans AS before_scans ON before_scans.id = before.scan_id
|
|
92
|
+
JOIN finding_occurrences AS after ON after.id = matches.after_occurrence_id
|
|
93
|
+
JOIN scans AS after_scans ON after_scans.id = after.scan_id
|
|
94
|
+
WHERE before_scans.target_id = after_scans.target_id
|
|
95
|
+
"""
|
|
96
|
+
):
|
|
97
|
+
before = group((match["target_id"], match["before_finding_id"]))
|
|
98
|
+
after = group((match["target_id"], match["after_finding_id"]))
|
|
99
|
+
if before != after:
|
|
100
|
+
parents[after] = before
|
|
101
|
+
|
|
102
|
+
latest_scan_by_target = dict(
|
|
103
|
+
connection.execute(
|
|
104
|
+
"SELECT target_id, id FROM scans "
|
|
105
|
+
"WHERE status = 'complete' ORDER BY started_at, id"
|
|
99
106
|
)
|
|
107
|
+
)
|
|
108
|
+
|
|
109
|
+
grouped: dict[tuple[str, str], list[sqlite3.Row]] = {}
|
|
110
|
+
for row in connection.execute(
|
|
111
|
+
"""
|
|
100
112
|
SELECT
|
|
101
|
-
|
|
113
|
+
occurrences.id AS occurrence_id,
|
|
114
|
+
occurrences.finding_id,
|
|
115
|
+
occurrences.severity,
|
|
116
|
+
occurrences.created_at,
|
|
117
|
+
scans.id AS scan_id,
|
|
118
|
+
scans.started_at AS scan_started_at,
|
|
119
|
+
scans.target_id,
|
|
120
|
+
targets.current_path AS target_path,
|
|
121
|
+
scans.scope,
|
|
122
|
+
MAX(scans.updated_at, COALESCE(triage.updated_at, '')) AS updated_at,
|
|
123
|
+
triage.status AS decision_status,
|
|
124
|
+
triage.close_reason,
|
|
125
|
+
triage.updated_at AS decision_updated_at,
|
|
102
126
|
occurrences.title,
|
|
103
127
|
occurrences.summary,
|
|
104
128
|
(
|
|
105
129
|
SELECT locations.relative_path
|
|
106
130
|
FROM finding_locations AS locations
|
|
107
|
-
WHERE locations.occurrence_id =
|
|
131
|
+
WHERE locations.occurrence_id = occurrences.id
|
|
108
132
|
ORDER BY
|
|
109
133
|
CASE WHEN locations.role = 'root_control' THEN 0 ELSE 1 END,
|
|
110
134
|
locations.sort_order
|
|
111
135
|
LIMIT 1
|
|
112
136
|
) AS location_path
|
|
113
|
-
FROM
|
|
114
|
-
JOIN
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
ORDER BY
|
|
118
|
-
CASE selected_findings.status WHEN 'open' THEN 0 ELSE 1 END,
|
|
119
|
-
CASE selected_findings.severity
|
|
120
|
-
WHEN 'critical' THEN 0
|
|
121
|
-
WHEN 'high' THEN 1
|
|
122
|
-
WHEN 'medium' THEN 2
|
|
123
|
-
WHEN 'low' THEN 3
|
|
124
|
-
WHEN 'informational' THEN 4
|
|
125
|
-
ELSE 5
|
|
126
|
-
END,
|
|
127
|
-
selected_findings.created_at DESC,
|
|
128
|
-
selected_findings.occurrence_id
|
|
137
|
+
FROM finding_occurrences AS occurrences
|
|
138
|
+
JOIN scans ON scans.id = occurrences.scan_id
|
|
139
|
+
JOIN security_targets AS targets ON targets.id = scans.target_id
|
|
140
|
+
LEFT JOIN finding_triage AS triage ON triage.occurrence_id = occurrences.id
|
|
129
141
|
""",
|
|
142
|
+
):
|
|
143
|
+
grouped.setdefault(group((row["target_id"], row["finding_id"])), []).append(row)
|
|
144
|
+
|
|
145
|
+
findings = []
|
|
146
|
+
for occurrences in grouped.values():
|
|
147
|
+
latest = max(occurrences, key=lambda row: (row["created_at"], row["occurrence_id"]))
|
|
148
|
+
decision = max(
|
|
149
|
+
(row for row in occurrences if row["decision_status"] is not None),
|
|
150
|
+
key=lambda row: (row["decision_updated_at"], row["occurrence_id"]),
|
|
151
|
+
default=None,
|
|
152
|
+
)
|
|
153
|
+
status = decision["decision_status"] if decision is not None else "open"
|
|
154
|
+
if (
|
|
155
|
+
status == "closed"
|
|
156
|
+
and decision["close_reason"] == "already_fixed"
|
|
157
|
+
and latest["created_at"] > decision["decision_updated_at"]
|
|
158
|
+
):
|
|
159
|
+
status = "open"
|
|
160
|
+
scans = sorted({(row["scan_started_at"], row["scan_id"]) for row in occurrences})
|
|
161
|
+
findings.append(
|
|
162
|
+
{
|
|
163
|
+
**dict(latest),
|
|
164
|
+
"confirmed_in_latest_scan": latest_scan_by_target.get(latest["target_id"])
|
|
165
|
+
== latest["scan_id"],
|
|
166
|
+
"known_since": scans[0][0],
|
|
167
|
+
"known_scan_ids": [scan_id for _, scan_id in scans],
|
|
168
|
+
"matched_finding_ids": sorted({row["finding_id"] for row in occurrences}),
|
|
169
|
+
"occurrence_count": len(occurrences),
|
|
170
|
+
"status": status,
|
|
171
|
+
"updated_at": max(
|
|
172
|
+
latest["updated_at"],
|
|
173
|
+
decision["decision_updated_at"] if decision is not None else "",
|
|
174
|
+
),
|
|
175
|
+
}
|
|
176
|
+
)
|
|
177
|
+
|
|
178
|
+
findings.sort(key=lambda finding: finding["occurrence_id"])
|
|
179
|
+
findings.sort(
|
|
180
|
+
key=lambda finding: (
|
|
181
|
+
finding["status"] == "open",
|
|
182
|
+
-scan_history.SEVERITY_ORDER.get(finding["severity"], 5),
|
|
183
|
+
finding["created_at"],
|
|
184
|
+
),
|
|
185
|
+
reverse=True,
|
|
130
186
|
)
|
|
187
|
+
yield from findings
|
|
131
188
|
|
|
132
189
|
|
|
133
190
|
def list_repositories(
|
|
@@ -22,8 +22,12 @@ Use the shared scan artifact path conventions in `../../references/scan-artifact
|
|
|
22
22
|
|
|
23
23
|
Read `../../references/security-guidance.md` and resolve the applicable policy before inspecting each source file. A delegated file-review worker must do the same before reading its assigned source.
|
|
24
24
|
|
|
25
|
+
### Compact Diff Workflow
|
|
26
|
+
|
|
27
|
+
When a running diff scan already supplies its file inventory through `list_codex_security_review_items`, review that inventory directly and record all candidates once with `record_codex_security_discovery_candidates`. Do not generate ranked worklists, per-finding ledgers, discovery receipts, or discovery reports. Skip the legacy workflow and artifact requirements below.
|
|
28
|
+
|
|
25
29
|
### Code Diff Workflow
|
|
26
|
-
|
|
30
|
+
For a targeted code diff without an existing compact inventory:
|
|
27
31
|
|
|
28
32
|
- Read `../security-scan/references/scan-artifacts-and-ledger.md`.
|
|
29
33
|
- Generate `rank_input.jsonl` deterministically from changed source-like files with `<python_command> <plugin_dir>/scripts/generate_rank_input.py make-diff-rank-input --repo <repo_root> --base <base> --mode revisions --head <head> --out <discovery_dir>/rank_input.jsonl` for PR, commit, and branch diffs, or `<python_command> <plugin_dir>/scripts/generate_rank_input.py make-diff-rank-input --repo <repo_root> --base <base> --mode local-patch --out <discovery_dir>/rank_input.jsonl` for a local patch.
|
|
@@ -137,7 +141,7 @@ Otherwise, for each candidate include:
|
|
|
137
141
|
- taxonomy with CWE IDs when known
|
|
138
142
|
- enough evidence that a later reviewer can understand why the candidate is technically plausible before validation
|
|
139
143
|
|
|
140
|
-
For diff-scoped discovery, when candidates are emitted, create the per-finding directory from `../../references/scan-artifacts.md` and append one discovery receipt to that finding's candidate ledger. The ledger row should identify the candidate, scan scope, discovery status, affected locations, and the discovery artifact or evidence that produced it.
|
|
144
|
+
For legacy diff-scoped discovery without a compact inventory, when candidates are emitted, create the per-finding directory from `../../references/scan-artifacts.md` and append one discovery receipt to that finding's candidate ledger. The ledger row should identify the candidate, scan scope, discovery status, affected locations, and the discovery artifact or evidence that produced it.
|
|
141
145
|
|
|
142
146
|
|
|
143
147
|
## Hard Rules
|
|
@@ -146,8 +150,8 @@ For diff-scoped discovery, when candidates are emitted, create the per-finding d
|
|
|
146
150
|
- Focus on the actual changes, not the commit message.
|
|
147
151
|
- Stay anchored to the diff and the files it relies on for diff-scoped scans.
|
|
148
152
|
- Candidate discovery is about plausibility, not final severity.
|
|
149
|
-
- For diff-scoped discovery, do not emit an untracked candidate. Every candidate finding needs a stable candidate id and a discovery receipt in its candidate-ledger path from `../../references/scan-artifacts.md` so later validation and attack-path analysis can prove coverage for that exact finding.
|
|
153
|
+
- For legacy diff-scoped discovery without a compact inventory, do not emit an untracked candidate. Every candidate finding needs a stable candidate id and a discovery receipt in its candidate-ledger path from `../../references/scan-artifacts.md` so later validation and attack-path analysis can prove coverage for that exact finding.
|
|
150
154
|
- Do not add `relevant_lines` when no bug exists. For diff-scoped scans, add `relevant_lines` only when the bug overlaps the diff and those lines are relevant to the bug.
|
|
151
155
|
- Do not turn discovery into full validation or full severity calibration.
|
|
152
156
|
- Continue reviewing until no additional distinct plausible candidates remain.
|
|
153
|
-
- For diff-scoped discovery, save a final visible report using the finding discovery report path from `../../references/scan-artifacts.md`.
|
|
157
|
+
- For legacy diff-scoped discovery without a compact inventory, save a final visible report using the finding discovery report path from `../../references/scan-artifacts.md`.
|