vouchington-tooling 0.72.1 → 0.72.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +27 -5
- package/dist/cli/commands/docker-image-mirror.mjs +7 -0
- package/dist/cli/usage-text-policy.d.mts +1 -1
- package/dist/cli/usage-text-policy.mjs +3 -0
- package/dist/cli/usage-text.d.mts +1 -1
- package/dist/docker-image-mirror/images.mjs +27 -16
- package/dist/docker-image-mirror/index.mjs +1 -0
- package/dist/docker-image-mirror/local-builds.d.mts +3 -1
- package/dist/docker-image-mirror/local-builds.mjs +47 -13
- package/dist/docker-image-mirror/policy.d.mts +5 -0
- package/dist/docker-image-mirror/policy.mjs +2 -0
- package/dist/docker-image-mirror/shell-vars.d.mts +24 -0
- package/dist/docker-image-mirror/shell-vars.mjs +62 -0
- package/package.json +1 -1
- package/scripts/__pycache__/run_bounded_process.cpython-312.pyc +0 -0
- package/skills/bounded-iteration/SKILL.md +2 -1
- package/skills/bounded-iteration/references/sweep-patterns.md +4 -0
- package/skills/manifest.json +4 -4
- package/skills/postgres-node-performance-tuning/SKILL.md +2 -1
- package/skills/postgres-node-performance-tuning/references/performance-patterns.md +22 -6
- package/skills/postgres-partitioning-uuid-v7/references/partition-lifecycle.md +5 -2
- package/skills/postgres-schema-design/SKILL.md +1 -0
- package/skills/postgres-schema-design/references/relations.md +3 -1
- package/skills/postgres-schema-design/references/table-shapes.md +31 -11
package/README.md
CHANGED
|
@@ -428,7 +428,7 @@ exact versioned `deb`/`apk`/`rpm` purls, a `Not exploitable: <reason>. Cleanup:
|
|
|
428
428
|
statement (the URL must start with `--cleanup-url-prefix` when given, else any GitHub issue) and an
|
|
429
429
|
`expired_at` UTC date after today and at most `--max-expiry-days` (default 30) away.
|
|
430
430
|
|
|
431
|
-
`vouchington docker-image-mirror [--mirror <host>] [--allow-registry <prefix>]... [--require-digest]
|
|
431
|
+
`vouchington docker-image-mirror [--mirror <host>] [--allow-registry <prefix>]... [--require-digest] [--allow-ref-expression <text>]...` reports every Docker Hub image reference in
|
|
432
432
|
tracked workflows, composite actions and Dockerfiles: `docker run|pull|create`, `container:`,
|
|
433
433
|
`services:`, `docker://` uses, `docker/setup-buildx-action` drivers, `FROM`, `COPY --from` and
|
|
434
434
|
`# syntax=`. An image is accepted when it names a registry host; `--mirror` (default
|
|
@@ -454,13 +454,35 @@ exempt when it exactly equals a tag that file produces, or is exactly
|
|
|
454
454
|
`${{ steps.<id>.outputs.<name> }}` where `<id>` is a step that builds. Produced tags are the
|
|
455
455
|
`-t`/`--tag`/`--tag=` values of `docker build`, `docker buildx build` and `podman build` in `run:`
|
|
456
456
|
steps, and the newline- or comma-separated `tags:` of a `docker/build-push-action` step with
|
|
457
|
-
`load: true`. A tag
|
|
458
|
-
|
|
459
|
-
|
|
457
|
+
`load: true`. A tag is recorded as normalized text: quotes are stripped, `${X}` and `${{ env.X }}`
|
|
458
|
+
become `$X`, and a variable the step's own `env:` defines (`REPO: ${{ inputs.repo }}`, `TAG: built`)
|
|
459
|
+
is replaced by its value, so each step's binding is compared by what it holds, never by name. A tag that is only
|
|
460
|
+
a variable is skipped. A `docker/metadata-action` `images:` name also counts as a built repository
|
|
461
|
+
(any tag, never a `@digest` form) when a `docker/build-push-action` (`load: true`) step's `tags:` or
|
|
462
|
+
a `docker/bake-action` step's `files:` consumes that step's outputs. A bake step must not set
|
|
463
|
+
`push: true`, `load: false` or a non-docker `set: *.output=`.
|
|
464
|
+
|
|
465
|
+
An image argument that is exactly `$VAR`, `"$VAR"` or `${VAR}` of `docker run|pull|create` is
|
|
466
|
+
resolved through the last `VAR=value` (or `export`/`local`/`readonly`) line before it in the same
|
|
467
|
+
`run:` block, then compared as above. Only unindented assignments are trusted: a variable that is
|
|
468
|
+
also assigned on an indented line (a possible branch), in another block, after the command, or by
|
|
469
|
+
`$(...)`, or a single-quoted `'$VAR'`, is not resolved. A docker command inside `$(...)` no longer keeps the trailing `)`.
|
|
470
|
+
A matching tag in a different file does not count, and every other reference is checked as before.
|
|
471
|
+
Without either flag nothing is exempt.
|
|
472
|
+
|
|
473
|
+
`--allow-ref-expression <text>` (repeatable) is an audited escape hatch for dynamic values a scan
|
|
474
|
+
cannot verify, such as a job output validated by a trusted job: an image reference whose raw text
|
|
475
|
+
exactly equals the value is accepted by both checks. It is an exact string match, never a pattern,
|
|
476
|
+
and an empty value exits 2. Spell out each value in the command so it is reviewed:
|
|
477
|
+
|
|
478
|
+
```bash
|
|
479
|
+
vouchington docker-image-mirror --allow-registry ghcr.io/vouchington/dependency- --require-digest \
|
|
480
|
+
--allow-ref-expression '${{ needs.prepare.outputs.postgres_image }}'
|
|
481
|
+
```
|
|
460
482
|
|
|
461
483
|
The checks are exported as `checkTrivyPolicy(root, trackedFiles, options)` from
|
|
462
484
|
`vouchington-tooling/trivy-policy` (plus the command parsers and `validateIgnoreRegistry`) and
|
|
463
|
-
`checkDockerImageMirrors(root, trackedFiles, { mirror, allowRegistries, requireDigest })` from `vouchington-tooling/docker-image-mirror`;
|
|
485
|
+
`checkDockerImageMirrors(root, trackedFiles, { mirror, allowRegistries, requireDigest, allowRefExpressions })` from `vouchington-tooling/docker-image-mirror`;
|
|
464
486
|
both return violation strings.
|
|
465
487
|
|
|
466
488
|
## Running the Storybook browser tests
|
|
@@ -21,6 +21,7 @@ export async function runDockerImageMirrorCommand(args, dependencies = {}) {
|
|
|
21
21
|
mirror: { type: 'string' },
|
|
22
22
|
'allow-registry': { type: 'string', multiple: true },
|
|
23
23
|
'require-digest': { type: 'boolean' },
|
|
24
|
+
'allow-ref-expression': { type: 'string', multiple: true },
|
|
24
25
|
},
|
|
25
26
|
}).values;
|
|
26
27
|
}
|
|
@@ -33,6 +34,11 @@ export async function runDockerImageMirrorCommand(args, dependencies = {}) {
|
|
|
33
34
|
stderr.write(`vouchington: --allow-registry needs a non-empty prefix\n${commandUsage('docker-image-mirror')}`);
|
|
34
35
|
return 2;
|
|
35
36
|
}
|
|
37
|
+
const allowRefExpressions = values['allow-ref-expression'];
|
|
38
|
+
if (allowRefExpressions?.includes('')) {
|
|
39
|
+
stderr.write(`vouchington: --allow-ref-expression needs a non-empty value\n${commandUsage('docker-image-mirror')}`);
|
|
40
|
+
return 2;
|
|
41
|
+
}
|
|
36
42
|
const root = resolve(values.root ?? process.cwd());
|
|
37
43
|
const context = await (dependencies.buildContext ?? buildSharedContext)(root);
|
|
38
44
|
if (!context.isInsideGitRepo) {
|
|
@@ -43,6 +49,7 @@ export async function runDockerImageMirrorCommand(args, dependencies = {}) {
|
|
|
43
49
|
mirror: values.mirror,
|
|
44
50
|
allowRegistries,
|
|
45
51
|
requireDigest: values['require-digest'],
|
|
52
|
+
allowRefExpressions,
|
|
46
53
|
});
|
|
47
54
|
if (errors.length === 0) {
|
|
48
55
|
stdout.write('Docker image mirror policy: ok\n');
|
|
@@ -1,2 +1,2 @@
|
|
|
1
1
|
export declare const POLICY_COMMANDS = " coverage-scope-check Check positive-threshold coverage rules against the Vitest scope\n trivy-policy Check tracked Trivy workflow scans and the exception registry\n docker-image-mirror Require tracked Docker images to come from a Docker Hub mirror\n";
|
|
2
|
-
export declare const POLICY_USAGE = "\ncoverage-scope-check [--rules <path>] [--scope-file <json>] [--include <glob>]... [--exclude <glob>]...\n Fails (exit 1) when a git-tracked file that a coverage rules file (default .coverage-rules.yml)\n gives a patch_coverage_min above 0, and its scope does not ignore, is outside the collector's\n scope. The collector scope is the repeated --include/--exclude globs plus, optionally, the\n include/exclude string arrays of a --scope-file JSON written from the Vitest coverage config.\n At least one include glob is required.\n\ntrivy-policy [--root <dir>] [--ignore-file <path>] [--findings-exit-code-env <NAME>] [--cleanup-url-prefix <url>] [--min-gates N] [--min-sboms N] [--max-expiry-days N]\n Reads git-tracked workflows, composite actions and shell scripts and prints each violation (exit 1).\n Every Trivy package scan must be an auditable CLI run block (no action, wrapper, substitution or\n background). Vulnerability gates need --scanners vuln, --pkg-types os, --severity CRITICAL,HIGH,\n --ignore-unfixed, --skip-db-update, --quiet, table output to a file, --ignorefile <ignore-file>\n (default .trivyignore.yaml) and --exit-code \"$<NAME>\" (default TRIVY_FINDINGS_EXIT_CODE).\n CycloneDX SBOM scans must be unfiltered with --exit-code 0. At least --min-gates gates and\n --min-sboms SBOM scans (default 2 each) must exist. The tracked ignore registry needs purl-exact OS\n packages, a \"Not exploitable: ... Cleanup: <issue URL>\" statement (URL must start with\n --cleanup-url-prefix when given) and an expired_at between tomorrow and --max-expiry-days (30).\n\ndocker-image-mirror [--root <dir>] [--mirror <host>] [--allow-registry <prefix>]... [--require-digest]\n Prints each Docker Hub image in git-tracked workflows, composite actions and Dockerfiles (exit 1):\n run, pull and create commands, containers, services, docker:// references, buildx drivers, FROM,\n COPY --from and # syntax. Images must name a registry host; --mirror (default mirror.gcr.io) is\n the host the message tells you to use. --allow-registry (repeatable) instead requires every image,\n including FROM and COPY --from, to start with one of the prefixes; a reference that expands a\n variable passes only when its literal text before the variable starts with a prefix. --require-digest\n requires @sha256: plus 64 lowercase hex characters. Stage aliases and scratch stay exempt.\n\n";
|
|
2
|
+
export declare const POLICY_USAGE = "\ncoverage-scope-check [--rules <path>] [--scope-file <json>] [--include <glob>]... [--exclude <glob>]...\n Fails (exit 1) when a git-tracked file that a coverage rules file (default .coverage-rules.yml)\n gives a patch_coverage_min above 0, and its scope does not ignore, is outside the collector's\n scope. The collector scope is the repeated --include/--exclude globs plus, optionally, the\n include/exclude string arrays of a --scope-file JSON written from the Vitest coverage config.\n At least one include glob is required.\n\ntrivy-policy [--root <dir>] [--ignore-file <path>] [--findings-exit-code-env <NAME>] [--cleanup-url-prefix <url>] [--min-gates N] [--min-sboms N] [--max-expiry-days N]\n Reads git-tracked workflows, composite actions and shell scripts and prints each violation (exit 1).\n Every Trivy package scan must be an auditable CLI run block (no action, wrapper, substitution or\n background). Vulnerability gates need --scanners vuln, --pkg-types os, --severity CRITICAL,HIGH,\n --ignore-unfixed, --skip-db-update, --quiet, table output to a file, --ignorefile <ignore-file>\n (default .trivyignore.yaml) and --exit-code \"$<NAME>\" (default TRIVY_FINDINGS_EXIT_CODE).\n CycloneDX SBOM scans must be unfiltered with --exit-code 0. At least --min-gates gates and\n --min-sboms SBOM scans (default 2 each) must exist. The tracked ignore registry needs purl-exact OS\n packages, a \"Not exploitable: ... Cleanup: <issue URL>\" statement (URL must start with\n --cleanup-url-prefix when given) and an expired_at between tomorrow and --max-expiry-days (30).\n\ndocker-image-mirror [--root <dir>] [--mirror <host>] [--allow-registry <prefix>]... [--require-digest]\n [--allow-ref-expression <text>]...\n Prints each Docker Hub image in git-tracked workflows, composite actions and Dockerfiles (exit 1):\n run, pull and create commands, containers, services, docker:// references, buildx drivers, FROM,\n COPY --from and # syntax. Images must name a registry host; --mirror (default mirror.gcr.io) is\n the host the message tells you to use. --allow-registry (repeatable) instead requires every image,\n including FROM and COPY --from, to start with one of the prefixes; a reference that expands a\n variable passes only when its literal text before the variable starts with a prefix. --require-digest\n requires @sha256: plus 64 lowercase hex characters. Stage aliases and scratch stay exempt.\n --allow-ref-expression (repeatable) is an audited escape hatch for dynamic job outputs: a reference\n whose raw text exactly equals the value is accepted. Images the same file builds locally are exempt.\n\n";
|
|
@@ -17,6 +17,7 @@ trivy-policy [--root <dir>] [--ignore-file <path>] [--findings-exit-code-env <NA
|
|
|
17
17
|
--cleanup-url-prefix when given) and an expired_at between tomorrow and --max-expiry-days (30).
|
|
18
18
|
|
|
19
19
|
docker-image-mirror [--root <dir>] [--mirror <host>] [--allow-registry <prefix>]... [--require-digest]
|
|
20
|
+
[--allow-ref-expression <text>]...
|
|
20
21
|
Prints each Docker Hub image in git-tracked workflows, composite actions and Dockerfiles (exit 1):
|
|
21
22
|
run, pull and create commands, containers, services, docker:// references, buildx drivers, FROM,
|
|
22
23
|
COPY --from and # syntax. Images must name a registry host; --mirror (default mirror.gcr.io) is
|
|
@@ -24,5 +25,7 @@ docker-image-mirror [--root <dir>] [--mirror <host>] [--allow-registry <prefix>]
|
|
|
24
25
|
including FROM and COPY --from, to start with one of the prefixes; a reference that expands a
|
|
25
26
|
variable passes only when its literal text before the variable starts with a prefix. --require-digest
|
|
26
27
|
requires @sha256: plus 64 lowercase hex characters. Stage aliases and scratch stay exempt.
|
|
28
|
+
--allow-ref-expression (repeatable) is an audited escape hatch for dynamic job outputs: a reference
|
|
29
|
+
whose raw text exactly equals the value is accepted. Images the same file builds locally are exempt.
|
|
27
30
|
|
|
28
31
|
`;
|
|
@@ -1 +1 @@
|
|
|
1
|
-
export declare const USAGE = "Usage: vouchington <command> [options]\n\nCommands:\n runner-port-policy Print or validate a runner port policy\n with-host-lock Run a command under a host-wide lock\n host-storage Check free space on caller-supplied paths\n gha-runtime-audit Audit successful GitHub Actions job runtimes\n require-up-to-date Require HEAD to include a fetched remote branch\n fresh-base Classify the worktree against a remote base branch\n gitleaks-directory-scan Scan a directory with Gitleaks\n ast-grep-examples Run AST-grep rule examples\n ast-grep-pack Print shipped unconditional AST-grep pack paths as JSON\n gha-workspace-policy Check GitHub Actions workspace safety policy\n gha-output Write a collision-safe multiline GITHUB_OUTPUT record\n gha-needs-results Fail if required GitHub Actions job results failed\n download-with-diagnostics Download a URL and report HTTP status on failure\n download-optional-run-artifacts Download optional artifacts from the current run\n host-pressure-diagnostics Print a bounded host memory/OOM/PSI snapshot\n allocate-browser-safe-ports Allocate Fetch-safe localhost ports\n diagnose-port-collision Capture bounded localhost port diagnostics\n prepare-trivy-db Download the Trivy vulnerability database\n gha-artifacts-cleanup Delete classified GitHub Actions artifacts\n agent-issue-labels Map changed paths to labels; create missing repository labels\n http-origin Validate an optional HTTP(S) origin\n vitest-blob-manifest Stamp a vitest-blob-manifest:v1 identity file\n vitest-report-attempt Write or read a Vitest report-attempt marker\n prepare-vitest-reports Validate and select Vitest report JSON files\n pnpm-install Install a pnpm workspace with retry and release-age fail-fast\n check-cache-size Measure a path and decide whether to save a GHA cache\n make-shard-matrix Emit a [1..N] GitHub Actions shard matrix\n load-runner-env Overlay a runner env file onto GITHUB_ENV with injection guards\n clean-workspace Reset a persistent-runner workspace with a fork-PR trust gate\n install-github-release Download a checksum-verified GitHub Release binary\n run-with-timeout Run a command with GNU timeout or a Perl fallback\n lint-links Two-pass lychee: internal links fail, external warn\n materialize-pr-context Dump PR title/body/files/diff/comments and #N crawl\n wait-for-apt-locks Wait until apt/dpkg lock files are free\n install-playwright-chromium-arm64 Install Playwright Chromium from browsers.json\n ghcr-package-retention Delete old GHCR package versions (KEEP_MIN or --main-history)\n harness-admission-lane Compute a GITHUB_RUN_ID admission lane for fleet fan-out\n harness-assert-gates Fail if any named HARNESS_*_ENABLED gate is enabled\n gha-collaborator-trust Classify authors as write-access collaborators or not\n nuget-central-version Validate a Directory.Packages.props PackageVersion delta\n verify-workflow-secrets Check a secret inventory against live repository and environment secrets\n dead-code Gate unused code against a committed jscpd baseline\n swift-semantic-equal Compare Swift sources ignoring comments and whitespace\n post-review Post one COMMENT review from a staged payload file\n stage-review-payload Validate a review payload file into a staging directory\n retrospective-transcript Format facts from Claude-compatible, Codex, or Grok transcripts\n link-skill Link one packaged skill into an explicit consumer directory\n retrospective-facts Gather immutable facts for a retrospective\n agent-blackboard Probe an Agent Blackboard deployment, classify distill sessions, manage snapshots\n mcp Serve the agent-blackboard journal tools over stdio (MCP)\n hook Run a repository policy module as a pre/post tool-use hook\n worktree Reclaim finished git worktrees (dry run unless --yes)\n tmux Run configured dev windows in tmux; name panes; agent reminders\n story-coverage Check component exports are imported by stories\n transient-retry Rerun a failed GitHub Actions run when an injected rule matches\n sandbox-command-audit Audit agent transcripts for sandbox escalations and denials\n cloc Count tracked lines of code by category and service with scc\n coverage-scope-check Check positive-threshold coverage rules against the Vitest scope\n trivy-policy Check tracked Trivy workflow scans and the exception registry\n docker-image-mirror Require tracked Docker images to come from a Docker Hub mirror\n storybook-browser-runner Run the Storybook Vitest browser project with cache, retry and watchdog\n\nOptions:\n -h, --help Show this help\n -v, --version Print the package version\n\nrunner-port-policy\n (no args) Print the shipped policy as JSON\n --file <path> Validate and print a policy file\n --reserved <port> Print true if the port is reserved\n\nwith-host-lock\n --name <family>\n [--slots <n>]\n --timeout-seconds <n>\n [--command-timeout-seconds <n>]\n [--failure-diagnostics <absolute-script>]\n [--on-acquire-timeout fail|run-unlocked]\n -- <command> [args...]\n\nhost-storage --path <path> [--path <path>...] [--minimum-free-bytes <bytes>] [--pnpm-store] [--docker-root] [--docker-desktop-path <path>] [--discovery-timeout-ms <ms>]\n\ngha-runtime-audit\n [--repository owner/name] Default GITHUB_REPOSITORY\n [--branch main]\n [--median-threshold-floor <seconds>] Off unless set. Must be below the ceiling\n [--median-threshold-ceiling <seconds>] Default 360. Flags a five-sample median above this\n --pr-workflow <name|/regex/> Repeatable\n --push-workflow <name|/regex/> Repeatable\n\nrequire-up-to-date --remote <name> --branch <name>\nfresh-base [--remote <name>] [--branch <name>] [--skip-fetch]\ngitleaks-directory-scan --config <path> [--directory <path>]\nast-grep-examples --rules <directory> --config <path>\nast-grep-pack\ngha-workspace-policy [--root <directory>] [--workflow-directory <directory>] [--action-directory <directory>]\n\ngha-output <name>\ngha-needs-results [label]\ndownload-with-diagnostics <url> <destination> [-- curl-args...]\ndownload-optional-run-artifacts (--name <name>... | --pattern <pattern>) --dir <directory>\nhost-pressure-diagnostics\nallocate-browser-safe-ports <count> [--policy path] [--forbidden-ports path]\ndiagnose-port-collision [--ports \"2200 2216\"] [--output-dir PATH]\nprepare-trivy-db\ngha-artifacts-cleanup run --run-id <id> [--keep-pattern glob] [--delete-pattern glob] [--patterns-file json]\ngha-artifacts-cleanup sweep --older-than-hours <n> [--keep-pattern glob] [--delete-pattern glob] [--patterns-file json]\nagent-issue-labels labels-from-paths [--labeler <path>] <path>...\nagent-issue-labels ensure-labels [--repo owner/name] [--policy-file json] [--update] <label>...\nhttp-origin [--field NAME] [value]\nvitest-blob-manifest <suite> [reports-directory]\nvitest-report-attempt <write DIRECTORY SUITE|read ROOT>\nprepare-vitest-reports [primary-directory] [fallback-directory] [output-directory]\npnpm-install --runner-lifecycle persistent|ephemeral|ephemeral-full --install-scripts true|false\ncheck-cache-size <path> <max-bytes> <label>\nmake-shard-matrix <total>\nload-runner-env\nclean-workspace\ninstall-github-release --repo owner/name --version X --asset 'name-{platform}.tar.gz' --bin name [--tag-prefix PREFIX] [--expected-sha256 SHA256] [--no-checksum] [--checksums-asset NAME] [--version-flag FLAG] [--bin-dir DIR]\nrun-with-timeout <timeout-seconds> <kill-after-seconds> <command...>\nlint-links [--offline] [--config PATH] [--glob PATTERN] [files...]\nmaterialize-pr-context\nwait-for-apt-locks\ninstall-playwright-chromium-arm64 [name:archive...]\nghcr-package-retention [--main-history] [--apply|--dry-run] [<url-encoded-package>...]\nharness-admission-lane <lanes>\nharness-assert-gates <gate>...\ngha-collaborator-trust <owner/repo> [--allow-bot <login>]... < authors.json\nnuget-central-version <trusted-props> <candidate-props> <metadata-json> <output-props>\nverify-workflow-secrets [--repo OWNER/NAME] [--confirm NAME,...] [--inventory MODULE] [--export NAME] [--branch NAME]\ndead-code --baseline PATH [--seed|--update] [--config PATH] [--binary PATH] [--repo-root PATH] [--entry GLOB]... [--snapshot-ignore GLOB]... [--exclude GLOB]...\nswift-semantic-equal <base> <head> <file.swift>\npost-review\nstage-review-payload optional|required <source> <destination>\nretrospective-transcript [--session-id ID] [--jsonl PATH] [--projects-dir PATH] [--codex-sessions-dir PATH] [--grok-sessions-dir PATH]\nlink-skill <name> --source-root <skills-dir> --target-root <consumer-skills-dir> Link a packaged or repository-local skill\nretrospective-facts (--pr NUMBER | --branch NAME | --no-pr) [--repo OWNER/NAME] [--raw]\nagent-blackboard probe\nagent-blackboard session-start [--skip-env NAME] [--setup-hint TEXT] [--docs-path PATH] [--restore-hint TEXT] [--timeout-ms N] (SessionStart hook; payload on stdin)\nagent-blackboard distill classify <partition.jsonl> --retro-cutoff ISO8601 --session-cutoff ISO8601\nagent-blackboard snapshot partition --snapshot PATH --checksum SHA256 --counts '{\"sessions\":N,\"entries\":N,\"records\":N,\"bytes\":N}'\nagent-blackboard snapshot cleanup [--snapshot PATH] [--partition-directory PATH --receipt JSON]\nmcp\n (no args) Serve journal_append, journal_entries, outbox_status, outbox_flush,\n session_ensure, retrospective_check, retrospective_save,\n pr_validate, pr_create, pr_update,\n snapshot_export, snapshot_cleanup, and session_archive over stdio.\n Needs the optional @modelcontextprotocol/sdk peer (and zod).\n Launch it from inside the repository's git worktree.\n\nhook pre-tool-use|post-tool-use --harness claude|codex|cursor|grok --policy <module> [--root <dir>] [--automation-env NAME]...\n Reads the harness hook payload on stdin. <module> default-exports a policy object (or a function\n of the hook context returning one) with a command rule table, post-tool-use checkers, and an\n optional reminder. pre-tool-use exits 2 to block, and also when the policy cannot load.\n post-tool-use is advisory and always exits 0. --root defaults to the working directory.\n Each --automation-env NAME (repeatable) marks the run unattended when that environment variable\n is set, non-empty, and not 0 or false; CI markers and claude -p are always unattended.\n\nworktree reclaim [--yes] [--json] [--max-worktrees N] [--timeout-ms N] [--pr-limit N] [--after PATH]\n Dry run by default. Removes clean, fully pushed worktrees whose same-repository PR for the\n branch is merged or closed at exactly HEAD, using git worktree remove (never forced).\n Skips the main checkout, the current worktree, and locked, bare or detached worktrees.\n\ntransient-retry --rules <module> [--repo owner/name] [--token-env NAME] [--dry-run] <run-id>\n Loads a module exporting \"rules\" (and optionally \"limits\"), evaluates them against the failed run,\n and reruns only on a match. Exits 0 on rerun, 2 when none is warranted, 1 on error.\n\nsandbox-command-audit --rules <module> [--settings-path <file>] [--repo-root <dir>]... [--worktree-glob <glob>]... [--projects-dir <dir>] [--codex-sessions-dir <dir>] [--session-id <id>] [--limit N] [--since DAYS] [--json] [--raw]\n Scans Claude and Codex transcripts scoped to this repository (git roots plus any --worktree-glob\n layouts) and reports Bash escalations, denials and permission failures by command prefix.\n <module> exports an object (default or \"rules\") with loadPolicy({ settingsPath, cwd }), which\n returns { covers(list, commandOrPrefix) } for the allow/deny/excluded lists or { error }, and\n optionally classifyFailure(command, text, source), limits and worktreeGlobs. Transcript roots\n default to CLAUDE_CONFIG_DIR/projects and CODEX_HOME/sessions. --raw prints full command text.\n\ncloc --config <file> [--scc-bin <path>] [--json]\n Counts git-tracked lines with scc (SCC_BIN or --scc-bin, else \"scc\"), classified into\n source/tests/tooling and by service, and prints an aligned table (--json: rows). <file> is a .json\n file or a module whose default (or \"config\") export is the layout: services [{ name, match }],\n defaultService, categories, testPatterns, toolingPatterns, countedRoots, countedRootFiles,\n excludedDirs, ignoredDirs, sccBin, missingSccHint.\n\ncoverage-scope-check [--rules <path>] [--scope-file <json>] [--include <glob>]... [--exclude <glob>]...\n Fails (exit 1) when a git-tracked file that a coverage rules file (default .coverage-rules.yml)\n gives a patch_coverage_min above 0, and its scope does not ignore, is outside the collector's\n scope. The collector scope is the repeated --include/--exclude globs plus, optionally, the\n include/exclude string arrays of a --scope-file JSON written from the Vitest coverage config.\n At least one include glob is required.\n\ntrivy-policy [--root <dir>] [--ignore-file <path>] [--findings-exit-code-env <NAME>] [--cleanup-url-prefix <url>] [--min-gates N] [--min-sboms N] [--max-expiry-days N]\n Reads git-tracked workflows, composite actions and shell scripts and prints each violation (exit 1).\n Every Trivy package scan must be an auditable CLI run block (no action, wrapper, substitution or\n background). Vulnerability gates need --scanners vuln, --pkg-types os, --severity CRITICAL,HIGH,\n --ignore-unfixed, --skip-db-update, --quiet, table output to a file, --ignorefile <ignore-file>\n (default .trivyignore.yaml) and --exit-code \"$<NAME>\" (default TRIVY_FINDINGS_EXIT_CODE).\n CycloneDX SBOM scans must be unfiltered with --exit-code 0. At least --min-gates gates and\n --min-sboms SBOM scans (default 2 each) must exist. The tracked ignore registry needs purl-exact OS\n packages, a \"Not exploitable: ... Cleanup: <issue URL>\" statement (URL must start with\n --cleanup-url-prefix when given) and an expired_at between tomorrow and --max-expiry-days (30).\n\ndocker-image-mirror [--root <dir>] [--mirror <host>] [--allow-registry <prefix>]... [--require-digest]\n Prints each Docker Hub image in git-tracked workflows, composite actions and Dockerfiles (exit 1):\n run, pull and create commands, containers, services, docker:// references, buildx drivers, FROM,\n COPY --from and # syntax. Images must name a registry host; --mirror (default mirror.gcr.io) is\n the host the message tells you to use. --allow-registry (repeatable) instead requires every image,\n including FROM and COPY --from, to start with one of the prefixes; a reference that expands a\n variable passes only when its literal text before the variable starts with a prefix. --require-digest\n requires @sha256: plus 64 lowercase hex characters. Stage aliases and scratch stay exempt.\n\nstorybook-browser-runner --project <name> [--command <cmd>] [--result-file <path>]... [--completion-file <path>] [--coverage-arg=<arg>]... [--optimize-deps-config <path>] [--env-prefix <PREFIX>] [--progress-prefix <text>] [-- <argv>...]\n Spawns <cmd> (default pnpm) in its own process group with <argv> (default \"exec vitest run\n --project <name>\") plus the coverage args (default --coverage) and supervises it: a Vite optimizer\n cache, retries on known Vite and browser start-up flakes, and stall, hang and session-budget\n watchdogs. --result-file paths are removed before each attempt and --completion-file is the one\n whose appearance is recorded. Tuned by <PREFIX>_STARTUP_ATTEMPTS, _OPTIMIZER_STALL_MS, _HANG_MS,\n _SESSION_BUDGET_MS, _DIAGNOSTICS_FILE, _PERSISTENT_CACHE, _PERSISTENT_CACHE_DIR and _COVERAGE\n (PREFIX defaults to STORYBOOK_BROWSER). Exits with the resolved test exit code.\n\ntmux up --config <file> [--root <dir>] [--no-attach] [-v|--verbose]\ntmux name [--socket PATH --pane %ID --worktree PATH] <name> | read-title | post-tool | agent-reminder <event>\n Environment: TMUX_TARGET_BIN, TMUX_TARGET_ROOT, TMUX_NAME_COMMAND, AGENT_TMUX_SOCKET/PANE/WORKTREE.\n\nstory-coverage --root <dir> --component-root <rel> --story-root <rel> --source-extension <ext> --component-extension <ext> --marker <regex> --story-pattern <regex> [--alias <prefix>=<path>] [--exclusions <json>] [--exclusions-label <text>]\n";
|
|
1
|
+
export declare const USAGE = "Usage: vouchington <command> [options]\n\nCommands:\n runner-port-policy Print or validate a runner port policy\n with-host-lock Run a command under a host-wide lock\n host-storage Check free space on caller-supplied paths\n gha-runtime-audit Audit successful GitHub Actions job runtimes\n require-up-to-date Require HEAD to include a fetched remote branch\n fresh-base Classify the worktree against a remote base branch\n gitleaks-directory-scan Scan a directory with Gitleaks\n ast-grep-examples Run AST-grep rule examples\n ast-grep-pack Print shipped unconditional AST-grep pack paths as JSON\n gha-workspace-policy Check GitHub Actions workspace safety policy\n gha-output Write a collision-safe multiline GITHUB_OUTPUT record\n gha-needs-results Fail if required GitHub Actions job results failed\n download-with-diagnostics Download a URL and report HTTP status on failure\n download-optional-run-artifacts Download optional artifacts from the current run\n host-pressure-diagnostics Print a bounded host memory/OOM/PSI snapshot\n allocate-browser-safe-ports Allocate Fetch-safe localhost ports\n diagnose-port-collision Capture bounded localhost port diagnostics\n prepare-trivy-db Download the Trivy vulnerability database\n gha-artifacts-cleanup Delete classified GitHub Actions artifacts\n agent-issue-labels Map changed paths to labels; create missing repository labels\n http-origin Validate an optional HTTP(S) origin\n vitest-blob-manifest Stamp a vitest-blob-manifest:v1 identity file\n vitest-report-attempt Write or read a Vitest report-attempt marker\n prepare-vitest-reports Validate and select Vitest report JSON files\n pnpm-install Install a pnpm workspace with retry and release-age fail-fast\n check-cache-size Measure a path and decide whether to save a GHA cache\n make-shard-matrix Emit a [1..N] GitHub Actions shard matrix\n load-runner-env Overlay a runner env file onto GITHUB_ENV with injection guards\n clean-workspace Reset a persistent-runner workspace with a fork-PR trust gate\n install-github-release Download a checksum-verified GitHub Release binary\n run-with-timeout Run a command with GNU timeout or a Perl fallback\n lint-links Two-pass lychee: internal links fail, external warn\n materialize-pr-context Dump PR title/body/files/diff/comments and #N crawl\n wait-for-apt-locks Wait until apt/dpkg lock files are free\n install-playwright-chromium-arm64 Install Playwright Chromium from browsers.json\n ghcr-package-retention Delete old GHCR package versions (KEEP_MIN or --main-history)\n harness-admission-lane Compute a GITHUB_RUN_ID admission lane for fleet fan-out\n harness-assert-gates Fail if any named HARNESS_*_ENABLED gate is enabled\n gha-collaborator-trust Classify authors as write-access collaborators or not\n nuget-central-version Validate a Directory.Packages.props PackageVersion delta\n verify-workflow-secrets Check a secret inventory against live repository and environment secrets\n dead-code Gate unused code against a committed jscpd baseline\n swift-semantic-equal Compare Swift sources ignoring comments and whitespace\n post-review Post one COMMENT review from a staged payload file\n stage-review-payload Validate a review payload file into a staging directory\n retrospective-transcript Format facts from Claude-compatible, Codex, or Grok transcripts\n link-skill Link one packaged skill into an explicit consumer directory\n retrospective-facts Gather immutable facts for a retrospective\n agent-blackboard Probe an Agent Blackboard deployment, classify distill sessions, manage snapshots\n mcp Serve the agent-blackboard journal tools over stdio (MCP)\n hook Run a repository policy module as a pre/post tool-use hook\n worktree Reclaim finished git worktrees (dry run unless --yes)\n tmux Run configured dev windows in tmux; name panes; agent reminders\n story-coverage Check component exports are imported by stories\n transient-retry Rerun a failed GitHub Actions run when an injected rule matches\n sandbox-command-audit Audit agent transcripts for sandbox escalations and denials\n cloc Count tracked lines of code by category and service with scc\n coverage-scope-check Check positive-threshold coverage rules against the Vitest scope\n trivy-policy Check tracked Trivy workflow scans and the exception registry\n docker-image-mirror Require tracked Docker images to come from a Docker Hub mirror\n storybook-browser-runner Run the Storybook Vitest browser project with cache, retry and watchdog\n\nOptions:\n -h, --help Show this help\n -v, --version Print the package version\n\nrunner-port-policy\n (no args) Print the shipped policy as JSON\n --file <path> Validate and print a policy file\n --reserved <port> Print true if the port is reserved\n\nwith-host-lock\n --name <family>\n [--slots <n>]\n --timeout-seconds <n>\n [--command-timeout-seconds <n>]\n [--failure-diagnostics <absolute-script>]\n [--on-acquire-timeout fail|run-unlocked]\n -- <command> [args...]\n\nhost-storage --path <path> [--path <path>...] [--minimum-free-bytes <bytes>] [--pnpm-store] [--docker-root] [--docker-desktop-path <path>] [--discovery-timeout-ms <ms>]\n\ngha-runtime-audit\n [--repository owner/name] Default GITHUB_REPOSITORY\n [--branch main]\n [--median-threshold-floor <seconds>] Off unless set. Must be below the ceiling\n [--median-threshold-ceiling <seconds>] Default 360. Flags a five-sample median above this\n --pr-workflow <name|/regex/> Repeatable\n --push-workflow <name|/regex/> Repeatable\n\nrequire-up-to-date --remote <name> --branch <name>\nfresh-base [--remote <name>] [--branch <name>] [--skip-fetch]\ngitleaks-directory-scan --config <path> [--directory <path>]\nast-grep-examples --rules <directory> --config <path>\nast-grep-pack\ngha-workspace-policy [--root <directory>] [--workflow-directory <directory>] [--action-directory <directory>]\n\ngha-output <name>\ngha-needs-results [label]\ndownload-with-diagnostics <url> <destination> [-- curl-args...]\ndownload-optional-run-artifacts (--name <name>... | --pattern <pattern>) --dir <directory>\nhost-pressure-diagnostics\nallocate-browser-safe-ports <count> [--policy path] [--forbidden-ports path]\ndiagnose-port-collision [--ports \"2200 2216\"] [--output-dir PATH]\nprepare-trivy-db\ngha-artifacts-cleanup run --run-id <id> [--keep-pattern glob] [--delete-pattern glob] [--patterns-file json]\ngha-artifacts-cleanup sweep --older-than-hours <n> [--keep-pattern glob] [--delete-pattern glob] [--patterns-file json]\nagent-issue-labels labels-from-paths [--labeler <path>] <path>...\nagent-issue-labels ensure-labels [--repo owner/name] [--policy-file json] [--update] <label>...\nhttp-origin [--field NAME] [value]\nvitest-blob-manifest <suite> [reports-directory]\nvitest-report-attempt <write DIRECTORY SUITE|read ROOT>\nprepare-vitest-reports [primary-directory] [fallback-directory] [output-directory]\npnpm-install --runner-lifecycle persistent|ephemeral|ephemeral-full --install-scripts true|false\ncheck-cache-size <path> <max-bytes> <label>\nmake-shard-matrix <total>\nload-runner-env\nclean-workspace\ninstall-github-release --repo owner/name --version X --asset 'name-{platform}.tar.gz' --bin name [--tag-prefix PREFIX] [--expected-sha256 SHA256] [--no-checksum] [--checksums-asset NAME] [--version-flag FLAG] [--bin-dir DIR]\nrun-with-timeout <timeout-seconds> <kill-after-seconds> <command...>\nlint-links [--offline] [--config PATH] [--glob PATTERN] [files...]\nmaterialize-pr-context\nwait-for-apt-locks\ninstall-playwright-chromium-arm64 [name:archive...]\nghcr-package-retention [--main-history] [--apply|--dry-run] [<url-encoded-package>...]\nharness-admission-lane <lanes>\nharness-assert-gates <gate>...\ngha-collaborator-trust <owner/repo> [--allow-bot <login>]... < authors.json\nnuget-central-version <trusted-props> <candidate-props> <metadata-json> <output-props>\nverify-workflow-secrets [--repo OWNER/NAME] [--confirm NAME,...] [--inventory MODULE] [--export NAME] [--branch NAME]\ndead-code --baseline PATH [--seed|--update] [--config PATH] [--binary PATH] [--repo-root PATH] [--entry GLOB]... [--snapshot-ignore GLOB]... [--exclude GLOB]...\nswift-semantic-equal <base> <head> <file.swift>\npost-review\nstage-review-payload optional|required <source> <destination>\nretrospective-transcript [--session-id ID] [--jsonl PATH] [--projects-dir PATH] [--codex-sessions-dir PATH] [--grok-sessions-dir PATH]\nlink-skill <name> --source-root <skills-dir> --target-root <consumer-skills-dir> Link a packaged or repository-local skill\nretrospective-facts (--pr NUMBER | --branch NAME | --no-pr) [--repo OWNER/NAME] [--raw]\nagent-blackboard probe\nagent-blackboard session-start [--skip-env NAME] [--setup-hint TEXT] [--docs-path PATH] [--restore-hint TEXT] [--timeout-ms N] (SessionStart hook; payload on stdin)\nagent-blackboard distill classify <partition.jsonl> --retro-cutoff ISO8601 --session-cutoff ISO8601\nagent-blackboard snapshot partition --snapshot PATH --checksum SHA256 --counts '{\"sessions\":N,\"entries\":N,\"records\":N,\"bytes\":N}'\nagent-blackboard snapshot cleanup [--snapshot PATH] [--partition-directory PATH --receipt JSON]\nmcp\n (no args) Serve journal_append, journal_entries, outbox_status, outbox_flush,\n session_ensure, retrospective_check, retrospective_save,\n pr_validate, pr_create, pr_update,\n snapshot_export, snapshot_cleanup, and session_archive over stdio.\n Needs the optional @modelcontextprotocol/sdk peer (and zod).\n Launch it from inside the repository's git worktree.\n\nhook pre-tool-use|post-tool-use --harness claude|codex|cursor|grok --policy <module> [--root <dir>] [--automation-env NAME]...\n Reads the harness hook payload on stdin. <module> default-exports a policy object (or a function\n of the hook context returning one) with a command rule table, post-tool-use checkers, and an\n optional reminder. pre-tool-use exits 2 to block, and also when the policy cannot load.\n post-tool-use is advisory and always exits 0. --root defaults to the working directory.\n Each --automation-env NAME (repeatable) marks the run unattended when that environment variable\n is set, non-empty, and not 0 or false; CI markers and claude -p are always unattended.\n\nworktree reclaim [--yes] [--json] [--max-worktrees N] [--timeout-ms N] [--pr-limit N] [--after PATH]\n Dry run by default. Removes clean, fully pushed worktrees whose same-repository PR for the\n branch is merged or closed at exactly HEAD, using git worktree remove (never forced).\n Skips the main checkout, the current worktree, and locked, bare or detached worktrees.\n\ntransient-retry --rules <module> [--repo owner/name] [--token-env NAME] [--dry-run] <run-id>\n Loads a module exporting \"rules\" (and optionally \"limits\"), evaluates them against the failed run,\n and reruns only on a match. Exits 0 on rerun, 2 when none is warranted, 1 on error.\n\nsandbox-command-audit --rules <module> [--settings-path <file>] [--repo-root <dir>]... [--worktree-glob <glob>]... [--projects-dir <dir>] [--codex-sessions-dir <dir>] [--session-id <id>] [--limit N] [--since DAYS] [--json] [--raw]\n Scans Claude and Codex transcripts scoped to this repository (git roots plus any --worktree-glob\n layouts) and reports Bash escalations, denials and permission failures by command prefix.\n <module> exports an object (default or \"rules\") with loadPolicy({ settingsPath, cwd }), which\n returns { covers(list, commandOrPrefix) } for the allow/deny/excluded lists or { error }, and\n optionally classifyFailure(command, text, source), limits and worktreeGlobs. Transcript roots\n default to CLAUDE_CONFIG_DIR/projects and CODEX_HOME/sessions. --raw prints full command text.\n\ncloc --config <file> [--scc-bin <path>] [--json]\n Counts git-tracked lines with scc (SCC_BIN or --scc-bin, else \"scc\"), classified into\n source/tests/tooling and by service, and prints an aligned table (--json: rows). <file> is a .json\n file or a module whose default (or \"config\") export is the layout: services [{ name, match }],\n defaultService, categories, testPatterns, toolingPatterns, countedRoots, countedRootFiles,\n excludedDirs, ignoredDirs, sccBin, missingSccHint.\n\ncoverage-scope-check [--rules <path>] [--scope-file <json>] [--include <glob>]... [--exclude <glob>]...\n Fails (exit 1) when a git-tracked file that a coverage rules file (default .coverage-rules.yml)\n gives a patch_coverage_min above 0, and its scope does not ignore, is outside the collector's\n scope. The collector scope is the repeated --include/--exclude globs plus, optionally, the\n include/exclude string arrays of a --scope-file JSON written from the Vitest coverage config.\n At least one include glob is required.\n\ntrivy-policy [--root <dir>] [--ignore-file <path>] [--findings-exit-code-env <NAME>] [--cleanup-url-prefix <url>] [--min-gates N] [--min-sboms N] [--max-expiry-days N]\n Reads git-tracked workflows, composite actions and shell scripts and prints each violation (exit 1).\n Every Trivy package scan must be an auditable CLI run block (no action, wrapper, substitution or\n background). Vulnerability gates need --scanners vuln, --pkg-types os, --severity CRITICAL,HIGH,\n --ignore-unfixed, --skip-db-update, --quiet, table output to a file, --ignorefile <ignore-file>\n (default .trivyignore.yaml) and --exit-code \"$<NAME>\" (default TRIVY_FINDINGS_EXIT_CODE).\n CycloneDX SBOM scans must be unfiltered with --exit-code 0. At least --min-gates gates and\n --min-sboms SBOM scans (default 2 each) must exist. The tracked ignore registry needs purl-exact OS\n packages, a \"Not exploitable: ... Cleanup: <issue URL>\" statement (URL must start with\n --cleanup-url-prefix when given) and an expired_at between tomorrow and --max-expiry-days (30).\n\ndocker-image-mirror [--root <dir>] [--mirror <host>] [--allow-registry <prefix>]... [--require-digest]\n [--allow-ref-expression <text>]...\n Prints each Docker Hub image in git-tracked workflows, composite actions and Dockerfiles (exit 1):\n run, pull and create commands, containers, services, docker:// references, buildx drivers, FROM,\n COPY --from and # syntax. Images must name a registry host; --mirror (default mirror.gcr.io) is\n the host the message tells you to use. --allow-registry (repeatable) instead requires every image,\n including FROM and COPY --from, to start with one of the prefixes; a reference that expands a\n variable passes only when its literal text before the variable starts with a prefix. --require-digest\n requires @sha256: plus 64 lowercase hex characters. Stage aliases and scratch stay exempt.\n --allow-ref-expression (repeatable) is an audited escape hatch for dynamic job outputs: a reference\n whose raw text exactly equals the value is accepted. Images the same file builds locally are exempt.\n\nstorybook-browser-runner --project <name> [--command <cmd>] [--result-file <path>]... [--completion-file <path>] [--coverage-arg=<arg>]... [--optimize-deps-config <path>] [--env-prefix <PREFIX>] [--progress-prefix <text>] [-- <argv>...]\n Spawns <cmd> (default pnpm) in its own process group with <argv> (default \"exec vitest run\n --project <name>\") plus the coverage args (default --coverage) and supervises it: a Vite optimizer\n cache, retries on known Vite and browser start-up flakes, and stall, hang and session-budget\n watchdogs. --result-file paths are removed before each attempt and --completion-file is the one\n whose appearance is recorded. Tuned by <PREFIX>_STARTUP_ATTEMPTS, _OPTIMIZER_STALL_MS, _HANG_MS,\n _SESSION_BUDGET_MS, _DIAGNOSTICS_FILE, _PERSISTENT_CACHE, _PERSISTENT_CACHE_DIR and _COVERAGE\n (PREFIX defaults to STORYBOOK_BROWSER). Exits with the resolved test exit code.\n\ntmux up --config <file> [--root <dir>] [--no-attach] [-v|--verbose]\ntmux name [--socket PATH --pane %ID --worktree PATH] <name> | read-title | post-tool | agent-reminder <event>\n Environment: TMUX_TARGET_BIN, TMUX_TARGET_ROOT, TMUX_NAME_COMMAND, AGENT_TMUX_SOCKET/PANE/WORKTREE.\n\nstory-coverage --root <dir> --component-root <rel> --story-root <rel> --source-extension <ext> --component-extension <ext> --marker <regex> --story-pattern <regex> [--alias <prefix>=<path>] [--exclusions <json>] [--exclusions-label <text>]\n";
|
|
@@ -1,16 +1,17 @@
|
|
|
1
1
|
import { withoutShellComment } from "../trivy-policy/shell.mjs";
|
|
2
2
|
import { collectLocalBuilds, isLocalBuild } from "./local-builds.mjs";
|
|
3
3
|
import { DEFAULT_MIRROR, cleanImage, imageReasons, policyActive, } from "./policy.mjs";
|
|
4
|
+
import { stepEnv, lastAssignment, normalizeText, variableName } from "./shell-vars.mjs";
|
|
4
5
|
export { DEFAULT_MIRROR } from "./policy.mjs";
|
|
5
6
|
// Unauthenticated Docker Hub pulls from hosted runners hit its rate limit. A pull-through mirror
|
|
6
7
|
// serves the same pinned digest without credentials, so every image must name a non-Hub registry.
|
|
7
8
|
// Shell scripts are not workflow bodies and are outside this scan.
|
|
8
9
|
const VALUE_FLAGS = new Set('-e --env -p --publish -v --volume --name --network --net --add-host --entrypoint -u --user -w --workdir --tmpfs --platform --mount --label -l --env-file --health-cmd --health-interval --health-retries --health-timeout --health-start-period -m --memory --cpus --restart --cap-add --cap-drop --shm-size --log-driver --ulimit -h --hostname --pull --pids-limit --group-add --device --dns --ipc --pid --cgroupns --stop-signal --stop-timeout --gpus --cidfile --link --volumes-from --cpuset-cpus --cpu-shares --cgroup-parent --dns-search --dns-option --ip --ip6 --mac-address --blkio-weight --memory-swap --memory-reservation --kernel-memory --oom-score-adj --runtime --security-opt --sysctl --userns --uts --annotation --network-alias --expose --label-file --isolation --log-opt --storage-opt --cpu-quota --cpu-period --domainname --detach-keys -a --attach --cpuset-mems --device-cgroup-rule'.split(' '));
|
|
9
10
|
const GLOBAL_VALUE_OPTIONS = new Set('--config --context -c --host -H --log-level -l --tlscacert --tlscert --tlskey'.split(' '));
|
|
10
|
-
function dockerCommandImages(script) {
|
|
11
|
+
function dockerCommandImages(script, env) {
|
|
11
12
|
const logical = withoutShellComment(script).replace(/\\\r?\n/gu, ' ');
|
|
12
|
-
return [...logical.matchAll(/\bdocker\s+([^\n;&|]*)/gu)].flatMap((
|
|
13
|
-
const tokens =
|
|
13
|
+
return [...logical.matchAll(/\bdocker\s+([^\n;&|]*)/gu)].flatMap((match) => {
|
|
14
|
+
const tokens = match[1].match(/(?:\$\{\{.*?\}\}|[^\s"')]|"[^"]*"|'[^']*')+/gu) ?? [];
|
|
14
15
|
let i = 0;
|
|
15
16
|
while (tokens[i]?.startsWith('-'))
|
|
16
17
|
i += GLOBAL_VALUE_OPTIONS.has(tokens[i]) ? 2 : 1;
|
|
@@ -21,13 +22,18 @@ function dockerCommandImages(script) {
|
|
|
21
22
|
for (i++; i < tokens.length; i++) {
|
|
22
23
|
const token = tokens[i];
|
|
23
24
|
if (!token.startsWith('-'))
|
|
24
|
-
return [token];
|
|
25
|
+
return [reference(token, logical.slice(0, match.index), env)];
|
|
25
26
|
if (VALUE_FLAGS.has(token))
|
|
26
27
|
i++;
|
|
27
28
|
}
|
|
28
29
|
return [];
|
|
29
30
|
});
|
|
30
31
|
}
|
|
32
|
+
function reference(raw, before, env) {
|
|
33
|
+
const name = variableName(raw);
|
|
34
|
+
const value = name === undefined ? undefined : lastAssignment(before, name);
|
|
35
|
+
return { raw, text: value ?? raw, env };
|
|
36
|
+
}
|
|
31
37
|
function stepImages(step) {
|
|
32
38
|
const driverOptions = step.with?.['driver-opts'];
|
|
33
39
|
const driverImage = /(?:^|[\s,])image=(\S+)/u.exec(typeof driverOptions === 'string' ? driverOptions : '');
|
|
@@ -36,33 +42,38 @@ function stepImages(step) {
|
|
|
36
42
|
const defaultsToHub = step.uses?.startsWith('docker/setup-buildx-action') === true &&
|
|
37
43
|
!driverImage &&
|
|
38
44
|
(driver === undefined || driver === 'docker-container');
|
|
39
|
-
|
|
40
|
-
...dockerCommandImages(step.run ?? ''),
|
|
45
|
+
const plain = [
|
|
41
46
|
...(driverImage ? [driverImage[1]] : []),
|
|
42
47
|
...(defaultsToHub ? ['moby/buildkit'] : []),
|
|
43
48
|
...(step.uses?.startsWith('docker://') ? [step.uses.slice('docker://'.length)] : []),
|
|
44
49
|
];
|
|
50
|
+
return [
|
|
51
|
+
...dockerCommandImages(step.run ?? '', stepEnv(step)),
|
|
52
|
+
...plain.map((raw) => ({ raw, text: raw })),
|
|
53
|
+
];
|
|
45
54
|
}
|
|
46
55
|
export function workflowImageViolations(path, document, mirror = DEFAULT_MIRROR, policy = {}) {
|
|
47
56
|
const found = [];
|
|
48
57
|
for (const [jobId, job] of Object.entries(document.jobs ?? {})) {
|
|
49
58
|
const container = typeof job.container === 'string' ? job.container : job.container?.image;
|
|
50
59
|
if (container)
|
|
51
|
-
found.push([`${jobId} container`, container]);
|
|
60
|
+
found.push([`${jobId} container`, { raw: container, text: container }]);
|
|
52
61
|
for (const [name, service] of Object.entries(job.services ?? {})) {
|
|
53
|
-
if (service.image)
|
|
54
|
-
found.push([`${jobId} service ${name}`, service.image]);
|
|
62
|
+
if (service.image) {
|
|
63
|
+
found.push([`${jobId} service ${name}`, { raw: service.image, text: service.image }]);
|
|
64
|
+
}
|
|
55
65
|
}
|
|
56
66
|
for (const step of job.steps ?? []) {
|
|
57
|
-
found.push(...stepImages(step).map((
|
|
67
|
+
found.push(...stepImages(step).map((ref) => [jobId, ref]));
|
|
58
68
|
}
|
|
59
69
|
}
|
|
60
70
|
const actionImage = document.runs?.image;
|
|
61
71
|
if (actionImage?.startsWith('docker://')) {
|
|
62
|
-
|
|
72
|
+
const image = actionImage.slice('docker://'.length);
|
|
73
|
+
found.push(['action image', { raw: image, text: image }]);
|
|
63
74
|
}
|
|
64
75
|
for (const step of document.runs?.steps ?? []) {
|
|
65
|
-
found.push(...stepImages(step).map((
|
|
76
|
+
found.push(...stepImages(step).map((ref) => ['composite', ref]));
|
|
66
77
|
}
|
|
67
78
|
// Images this file builds locally are never pulled, so the allowlist and digest pin skip them.
|
|
68
79
|
const builds = policyActive(policy)
|
|
@@ -71,11 +82,11 @@ export function workflowImageViolations(path, document, mirror = DEFAULT_MIRROR,
|
|
|
71
82
|
...(document.runs?.steps ?? []),
|
|
72
83
|
])
|
|
73
84
|
: undefined;
|
|
74
|
-
return found.flatMap(([where,
|
|
75
|
-
if (builds && isLocalBuild(cleanImage(
|
|
85
|
+
return found.flatMap(([where, { raw, text, env }]) => {
|
|
86
|
+
if (builds && isLocalBuild(normalizeText(cleanImage(text), env), builds))
|
|
76
87
|
return [];
|
|
77
|
-
const reasons = imageReasons(
|
|
78
|
-
return reasons.length > 0 ? [`${path}#${where}: ${
|
|
88
|
+
const reasons = imageReasons(raw, mirror, policy);
|
|
89
|
+
return reasons.length > 0 ? [`${path}#${where}: ${raw} ${reasons.join('; ')}`] : [];
|
|
79
90
|
});
|
|
80
91
|
}
|
|
81
92
|
export function dockerfileViolations(path, source, policy = {}, mirror = DEFAULT_MIRROR) {
|
|
@@ -25,6 +25,7 @@ export function checkDockerImageMirrors(root, trackedFiles, options = {}) {
|
|
|
25
25
|
const policy = {
|
|
26
26
|
allowRegistries: options.allowRegistries,
|
|
27
27
|
requireDigest: options.requireDigest,
|
|
28
|
+
allowRefExpressions: options.allowRefExpressions,
|
|
28
29
|
};
|
|
29
30
|
const errors = [];
|
|
30
31
|
for (const path of trackedFiles) {
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
export type BuildStep = {
|
|
2
2
|
id?: string;
|
|
3
|
+
env?: unknown;
|
|
3
4
|
run?: string;
|
|
4
5
|
uses?: string;
|
|
5
6
|
with?: Record<string, unknown>;
|
|
@@ -7,7 +8,8 @@ export type BuildStep = {
|
|
|
7
8
|
export interface LocalBuilds {
|
|
8
9
|
readonly tags: ReadonlySet<string>;
|
|
9
10
|
readonly stepIds: ReadonlySet<string>;
|
|
11
|
+
readonly repositories: ReadonlySet<string>;
|
|
10
12
|
}
|
|
11
13
|
export declare function collectLocalBuilds(steps: readonly BuildStep[]): LocalBuilds;
|
|
12
|
-
/** Whether a
|
|
14
|
+
/** Whether a normalized image reference is a locally built tag, repository or step output. */
|
|
13
15
|
export declare function isLocalBuild(image: string, builds: LocalBuilds): boolean;
|
|
@@ -1,10 +1,33 @@
|
|
|
1
1
|
import { withoutShellComment } from "../trivy-policy/shell.mjs";
|
|
2
|
+
import { normalizeText, stepEnv } from "./shell-vars.mjs";
|
|
2
3
|
const STEP_OUTPUT = /^\$\{\{\s*steps\.([\w-]+)\.outputs\.[\w-]+\s*\}\}$/u;
|
|
3
4
|
// A build counts only at a command position, not as text inside `echo` or another argument.
|
|
4
5
|
const BUILD_COMMAND = /(?:^|[;&|(]|\b(?:sudo|then|do|else))[ \t]*(?:docker\s+(?:buildx\s+)?|podman\s+)build\b([^\n;&|]*)/gmu;
|
|
5
|
-
function
|
|
6
|
-
const value = tag.trim()
|
|
7
|
-
return value === '' ||
|
|
6
|
+
function recorded(tag, step) {
|
|
7
|
+
const value = normalizeText(tag.trim(), stepEnv(step));
|
|
8
|
+
return value === '' || /^\$\w+$|^\$\{\{.*\}\}$/u.test(value) ? undefined : value;
|
|
9
|
+
}
|
|
10
|
+
/** Repository part of an image reference: everything before a trailing `:tag`. */
|
|
11
|
+
function repositoryOf(image) {
|
|
12
|
+
const colon = image.lastIndexOf(':');
|
|
13
|
+
return colon > image.lastIndexOf('/') ? image.slice(0, colon) : image;
|
|
14
|
+
}
|
|
15
|
+
function isLoaded(value) {
|
|
16
|
+
return value === true || value === 'true';
|
|
17
|
+
}
|
|
18
|
+
function isNotLoaded(value) {
|
|
19
|
+
return value === false || value === 'false';
|
|
20
|
+
}
|
|
21
|
+
/** Bake output overrides that keep the image out of the local store (cache-only, registry, ...). */
|
|
22
|
+
const FOREIGN_OUTPUT = /output=(?!(?:type=)?docker\b)/u;
|
|
23
|
+
/** A `docker/build-push-action` with `load: true`, or a `docker/bake-action` that does not push. */
|
|
24
|
+
function isBuildAction(step) {
|
|
25
|
+
if (step.uses?.startsWith('docker/bake-action@')) {
|
|
26
|
+
const set = step.with?.set;
|
|
27
|
+
const foreign = typeof set === 'string' && FOREIGN_OUTPUT.test(set);
|
|
28
|
+
return !isLoaded(step.with?.push) && !isNotLoaded(step.with?.load) && !foreign;
|
|
29
|
+
}
|
|
30
|
+
return step.uses?.startsWith('docker/build-push-action@') === true && isLoaded(step.with?.load);
|
|
8
31
|
}
|
|
9
32
|
/** Raw tag values of every build command in a script, or undefined when it builds nothing. */
|
|
10
33
|
function runBuildTags(script) {
|
|
@@ -14,7 +37,7 @@ function runBuildTags(script) {
|
|
|
14
37
|
return undefined;
|
|
15
38
|
const tags = [];
|
|
16
39
|
for (const [, args] of commands) {
|
|
17
|
-
const tokens = args.match(/(
|
|
40
|
+
const tokens = args.match(/(?:\$\{\{.*?\}\}|[^\s"']|"[^"]*"|'[^']*')+/gu) ?? [];
|
|
18
41
|
tokens.forEach((token, i) => {
|
|
19
42
|
if (token === '-t' || token === '--tag')
|
|
20
43
|
tags.push(tokens[i + 1] ?? '');
|
|
@@ -24,36 +47,47 @@ function runBuildTags(script) {
|
|
|
24
47
|
}
|
|
25
48
|
return tags;
|
|
26
49
|
}
|
|
27
|
-
/** Raw `tags:` values of a `docker/build-push-action` step
|
|
50
|
+
/** Raw `tags:` values of a built `docker/build-push-action` step. */
|
|
28
51
|
function actionBuildTags(step) {
|
|
29
|
-
|
|
30
|
-
if (!step.uses?.startsWith('docker/build-push-action@') || (load !== true && load !== 'true')) {
|
|
52
|
+
if (!step.uses?.startsWith('docker/build-push-action@') || !isBuildAction(step))
|
|
31
53
|
return undefined;
|
|
32
|
-
}
|
|
33
54
|
const tags = step.with?.tags;
|
|
34
55
|
return typeof tags === 'string' ? tags.split(/[\n,]/u) : [];
|
|
35
56
|
}
|
|
36
57
|
export function collectLocalBuilds(steps) {
|
|
37
58
|
const tags = new Set();
|
|
38
59
|
const stepIds = new Set();
|
|
60
|
+
const repositories = new Set();
|
|
61
|
+
const metadata = new Map();
|
|
39
62
|
for (const step of steps) {
|
|
63
|
+
if (step.id && step.uses?.startsWith('docker/metadata-action@')) {
|
|
64
|
+
const images = typeof step.with?.images === 'string' ? step.with.images.split(/[\n,]/u) : [];
|
|
65
|
+
metadata.set(step.id, images.flatMap((image) => recorded(image, step) ?? []));
|
|
66
|
+
}
|
|
40
67
|
const built = runBuildTags(step.run ?? '') ?? actionBuildTags(step);
|
|
41
|
-
if (built === undefined)
|
|
68
|
+
if (built === undefined && !isBuildAction(step))
|
|
42
69
|
continue;
|
|
43
70
|
if (step.id)
|
|
44
71
|
stepIds.add(step.id);
|
|
45
|
-
for (const tag of built) {
|
|
46
|
-
const value =
|
|
72
|
+
for (const tag of built ?? []) {
|
|
73
|
+
const value = recorded(tag, step);
|
|
47
74
|
if (value)
|
|
48
75
|
tags.add(value);
|
|
49
76
|
}
|
|
77
|
+
const used = [step.with?.tags, step.with?.files].filter((v) => typeof v === 'string').join('\n');
|
|
78
|
+
for (const [id, images] of metadata) {
|
|
79
|
+
if (used.includes(`steps.${id}.outputs.`))
|
|
80
|
+
images.forEach((image) => repositories.add(image));
|
|
81
|
+
}
|
|
50
82
|
}
|
|
51
|
-
return { tags, stepIds };
|
|
83
|
+
return { tags, stepIds, repositories };
|
|
52
84
|
}
|
|
53
|
-
/** Whether a
|
|
85
|
+
/** Whether a normalized image reference is a locally built tag, repository or step output. */
|
|
54
86
|
export function isLocalBuild(image, builds) {
|
|
55
87
|
if (builds.tags.has(image))
|
|
56
88
|
return true;
|
|
89
|
+
if (!image.includes('@') && builds.repositories.has(repositoryOf(image)))
|
|
90
|
+
return true;
|
|
57
91
|
const id = STEP_OUTPUT.exec(image)?.[1];
|
|
58
92
|
return id !== undefined && builds.stepIds.has(id);
|
|
59
93
|
}
|
|
@@ -4,6 +4,11 @@ export interface ImagePolicy {
|
|
|
4
4
|
readonly allowRegistries?: readonly string[] | undefined;
|
|
5
5
|
/** Require `@sha256:` followed by 64 lowercase hex characters. */
|
|
6
6
|
readonly requireDigest?: boolean | undefined;
|
|
7
|
+
/**
|
|
8
|
+
* Audited escape hatch for dynamic values (such as job outputs) a scan cannot verify: an image
|
|
9
|
+
* reference whose raw text exactly equals one of these is accepted. No patterns.
|
|
10
|
+
*/
|
|
11
|
+
readonly allowRefExpressions?: readonly string[] | undefined;
|
|
7
12
|
}
|
|
8
13
|
export declare function policyActive(policy: ImagePolicy): boolean;
|
|
9
14
|
export declare function cleanImage(reference: string): string;
|
|
@@ -34,6 +34,8 @@ function isHubImage(image) {
|
|
|
34
34
|
}
|
|
35
35
|
/** Reasons (empty when acceptable) that an image reference breaks the policy. */
|
|
36
36
|
export function imageReasons(reference, mirror, policy = {}) {
|
|
37
|
+
if (policy.allowRefExpressions?.includes(reference))
|
|
38
|
+
return [];
|
|
37
39
|
const image = cleanImage(reference);
|
|
38
40
|
if (image === '' || image === 'scratch')
|
|
39
41
|
return [];
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
export type EnvMap = ReadonlyMap<string, string>;
|
|
2
|
+
/**
|
|
3
|
+
* `envMapping` of a step, minus every variable its `run:` script assigns: a reassigned variable no
|
|
4
|
+
* longer holds the step's declared value, so it is left unsubstituted.
|
|
5
|
+
*/
|
|
6
|
+
export declare function stepEnv(step: {
|
|
7
|
+
env?: unknown;
|
|
8
|
+
run?: string;
|
|
9
|
+
}): EnvMap;
|
|
10
|
+
/**
|
|
11
|
+
* Strips quotes, rewrites `${X}` as `$X` and `${{ env.X }}` as `$X`, then replaces each `$NAME`
|
|
12
|
+
* the step `env:` defines with that value, so a step-local binding is compared by what it holds
|
|
13
|
+
* (an expression or a literal), not by its name.
|
|
14
|
+
*/
|
|
15
|
+
export declare function normalizeText(text: string, env?: EnvMap): string;
|
|
16
|
+
/** The variable name when an image argument is exactly `$VAR`, `"$VAR"` or `${VAR}` (not single-quoted). */
|
|
17
|
+
export declare function variableName(token: string): string | undefined;
|
|
18
|
+
/**
|
|
19
|
+
* Value of the last `VAR=value` statement in `before` (the same run block, earlier lines). Only
|
|
20
|
+
* unindented statements outside any open `(`/`{` group are trusted: an indented assignment may sit
|
|
21
|
+
* in a branch that does not run, and a grouped one in a subshell or function, so the variable is
|
|
22
|
+
* then left unresolved.
|
|
23
|
+
*/
|
|
24
|
+
export declare function lastAssignment(before: string, name: string): string | undefined;
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
// Text normalization shared by build tags and `docker run` image references, so a shell variable
|
|
2
|
+
// and the workflow expression feeding it compare equal. Matching stays exact: nothing here expands
|
|
3
|
+
// a value, it only rewrites spellings of the same name.
|
|
4
|
+
const EXPRESSION = /\$\{\{\s*([^}]*?)\s*\}\}/gu;
|
|
5
|
+
const SHELL_VARIABLE = /^\$(?:(\w+)|\{(\w+)\})$/u;
|
|
6
|
+
/**
|
|
7
|
+
* `envMapping` of a step, minus every variable its `run:` script assigns: a reassigned variable no
|
|
8
|
+
* longer holds the step's declared value, so it is left unsubstituted.
|
|
9
|
+
*/
|
|
10
|
+
export function stepEnv(step) {
|
|
11
|
+
const map = new Map(envMapping(step.env));
|
|
12
|
+
for (const [, name] of (step.run ?? '').matchAll(/(?:^|[\s;&|(])(\w+)=/gu))
|
|
13
|
+
map.delete(name);
|
|
14
|
+
return map;
|
|
15
|
+
}
|
|
16
|
+
/** Canonical text of each variable a step `env:` mapping defines, keyed by name. */
|
|
17
|
+
function envMapping(env) {
|
|
18
|
+
const map = new Map();
|
|
19
|
+
if (typeof env !== 'object' || env === null)
|
|
20
|
+
return map;
|
|
21
|
+
for (const [name, value] of Object.entries(env))
|
|
22
|
+
map.set(name, canonical(String(value).trim()));
|
|
23
|
+
return map;
|
|
24
|
+
}
|
|
25
|
+
function canonical(text) {
|
|
26
|
+
return text
|
|
27
|
+
.replace(/["']/gu, '')
|
|
28
|
+
.replace(EXPRESSION, (_, expression) => {
|
|
29
|
+
const name = /^env\.(\w+)$/u.exec(expression)?.[1];
|
|
30
|
+
return name ? `$${name}` : `\${{ ${expression} }}`;
|
|
31
|
+
})
|
|
32
|
+
.replace(/\$\{(\w+)\}(?!\w)/gu, '$$$1');
|
|
33
|
+
}
|
|
34
|
+
/**
|
|
35
|
+
* Strips quotes, rewrites `${X}` as `$X` and `${{ env.X }}` as `$X`, then replaces each `$NAME`
|
|
36
|
+
* the step `env:` defines with that value, so a step-local binding is compared by what it holds
|
|
37
|
+
* (an expression or a literal), not by its name.
|
|
38
|
+
*/
|
|
39
|
+
export function normalizeText(text, env = new Map()) {
|
|
40
|
+
return canonical(text).replace(/\$(\w+)/gu, (match, name) => env.get(name) ?? match);
|
|
41
|
+
}
|
|
42
|
+
/** The variable name when an image argument is exactly `$VAR`, `"$VAR"` or `${VAR}` (not single-quoted). */
|
|
43
|
+
export function variableName(token) {
|
|
44
|
+
const match = SHELL_VARIABLE.exec(token.replace(/^"(.*)"$/u, '$1'));
|
|
45
|
+
return match?.[1] ?? match?.[2];
|
|
46
|
+
}
|
|
47
|
+
/**
|
|
48
|
+
* Value of the last `VAR=value` statement in `before` (the same run block, earlier lines). Only
|
|
49
|
+
* unindented statements outside any open `(`/`{` group are trusted: an indented assignment may sit
|
|
50
|
+
* in a branch that does not run, and a grouped one in a subshell or function, so the variable is
|
|
51
|
+
* then left unresolved.
|
|
52
|
+
*/
|
|
53
|
+
export function lastAssignment(before, name) {
|
|
54
|
+
const pattern = new RegExp(`^([ \\t]*)(?:(?:export|local|readonly)[ \\t]+)?${name}=((?:"[^"]*"|'[^']*'|[^\\s;&|)"'])*)(?=[ \\t]*(?:$|[;&|)]))`, 'gmu');
|
|
55
|
+
const found = [...before.matchAll(pattern)];
|
|
56
|
+
const last = found.at(-1);
|
|
57
|
+
if (!last || found.some((match) => match[1] !== ''))
|
|
58
|
+
return undefined;
|
|
59
|
+
const prefix = before.slice(0, last.index).replace(/"[^"]*"|'[^']*'/gu, '');
|
|
60
|
+
const balance = (prefix.match(/[({]/gu)?.length ?? 0) - (prefix.match(/[)}]/gu)?.length ?? 0);
|
|
61
|
+
return balance === 0 ? last[2] : undefined;
|
|
62
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "vouchington-tooling",
|
|
3
|
-
"version": "0.72.
|
|
3
|
+
"version": "0.72.3",
|
|
4
4
|
"description": "Vouchington CLI and extractable tooling libraries.",
|
|
5
5
|
"homepage": "https://github.com/vouchington/vouchington-tooling/tree/main/packages/vouchington-tooling#readme",
|
|
6
6
|
"bugs": {
|
|
Binary file
|
|
@@ -29,7 +29,8 @@ A run's cost is the rows it reads and the work it starts, not the memory it hold
|
|
|
29
29
|
per run.
|
|
30
30
|
5. **Resume without restarting.** Keep a keyset position on the UUIDv7 key or a
|
|
31
31
|
`<verb>_through_at` time position, fix the sweep's end when it starts, and rescan an overlap
|
|
32
|
-
when insert order differs from commit order.
|
|
32
|
+
when insert order differs from commit order. Prefer the position in job data; a cursor table is
|
|
33
|
+
for a position that must be transactional with domain rows, and follows
|
|
33
34
|
[table shapes](../postgres-schema-design/references/table-shapes.md).
|
|
34
35
|
(`postgres-generated-column-predicates`)
|
|
35
36
|
6. **No unbounded fan-out.** Use `Promise.all` only over a bounded result, with bounded
|
|
@@ -81,6 +81,10 @@ parent id, `WHERE account_id = $1`, for the one account it was started for.
|
|
|
81
81
|
|
|
82
82
|
## High-water-mark window with an overlap
|
|
83
83
|
|
|
84
|
+
Prefer keeping the position and the sweep's fixed end in job data for a single-writer continuation.
|
|
85
|
+
Use a persisted cursor table only when the position must be transactional with domain rows, as
|
|
86
|
+
here.
|
|
87
|
+
|
|
84
88
|
An export reads `invoices` added since the last sweep. The cursor row holds the position and the
|
|
85
89
|
sweep's fixed end, and the job seeds its own row:
|
|
86
90
|
|
package/skills/manifest.json
CHANGED
|
@@ -30,7 +30,7 @@
|
|
|
30
30
|
{
|
|
31
31
|
"name": "bounded-iteration",
|
|
32
32
|
"plugin": "vouchington-database",
|
|
33
|
-
"pluginVersion": "0.
|
|
33
|
+
"pluginVersion": "0.8.0",
|
|
34
34
|
"path": "bounded-iteration/SKILL.md"
|
|
35
35
|
},
|
|
36
36
|
{
|
|
@@ -115,19 +115,19 @@
|
|
|
115
115
|
{
|
|
116
116
|
"name": "postgres-node-performance-tuning",
|
|
117
117
|
"plugin": "vouchington-database",
|
|
118
|
-
"pluginVersion": "0.
|
|
118
|
+
"pluginVersion": "0.8.0",
|
|
119
119
|
"path": "postgres-node-performance-tuning/SKILL.md"
|
|
120
120
|
},
|
|
121
121
|
{
|
|
122
122
|
"name": "postgres-partitioning-uuid-v7",
|
|
123
123
|
"plugin": "vouchington-database",
|
|
124
|
-
"pluginVersion": "0.
|
|
124
|
+
"pluginVersion": "0.8.0",
|
|
125
125
|
"path": "postgres-partitioning-uuid-v7/SKILL.md"
|
|
126
126
|
},
|
|
127
127
|
{
|
|
128
128
|
"name": "postgres-schema-design",
|
|
129
129
|
"plugin": "vouchington-database",
|
|
130
|
-
"pluginVersion": "0.
|
|
130
|
+
"pluginVersion": "0.8.0",
|
|
131
131
|
"path": "postgres-schema-design/SKILL.md"
|
|
132
132
|
},
|
|
133
133
|
{
|
|
@@ -18,7 +18,8 @@ Measure the query plan and the workload shape before changing code.
|
|
|
18
18
|
- Keep connection-pool usage bounded.
|
|
19
19
|
- Validate a query change at representative cardinality. Watch latency, memory, lock time, and
|
|
20
20
|
connection pressure together.
|
|
21
|
-
- Cut round trips: fold dependent reads, batch per-item lookups, and
|
|
21
|
+
- Cut round trips without weakening correctness: fold dependent reads, batch per-item lookups, and
|
|
22
|
+
overlap independent reads; see
|
|
22
23
|
[round trips](references/performance-patterns.md#round-trips).
|
|
23
24
|
- Read the plan for hidden per-row work before trusting a change; see
|
|
24
25
|
[reading plans](references/reading-plans.md).
|
|
@@ -3,10 +3,16 @@
|
|
|
3
3
|
Use pools and release checked-out clients in `finally`, especially for cursors, streams, and COPY.
|
|
4
4
|
Route ordinary reads to a replica when its lag is acceptable; route read-after-write, locking, and
|
|
5
5
|
transaction-consistent reads to the writer. Do not hold a client across unrelated application work.
|
|
6
|
+
Never hold a transaction, connection, or advisory lock across an external call; take a token lease
|
|
7
|
+
in a shared store instead, with a TTL, renewal, and compare-and-delete release. Stop work when
|
|
8
|
+
renewal fails, and guard any write a stale owner could make with the token.
|
|
6
9
|
|
|
7
10
|
For large reads, use keyset pagination, cursors, or streams with cancellation and bounded batches.
|
|
8
11
|
For writes, prefer set-based batches such as UNNEST or COPY when they preserve validation and error
|
|
9
|
-
handling.
|
|
12
|
+
handling. Do large list diffs and dedupes in the worker as a streaming comparison of sorted
|
|
13
|
+
inputs, not as writer-side set operations. Measure query plans with representative cardinality
|
|
14
|
+
before changing an index or query. Adding an index for a measured plan is a default: do not ask
|
|
15
|
+
first and do not make it a spec stop condition.
|
|
10
16
|
|
|
11
17
|
Check query predicates, joins, ordering, and selected columns against index shape. Use EXPLAIN
|
|
12
18
|
evidence to confirm planner behavior. Add extended statistics only when observed estimates show a
|
|
@@ -126,13 +132,23 @@ lateral subquery, rather than reading the parent and then the relation.
|
|
|
126
132
|
`Promise.all` over statements on one transaction client is still serial, because one connection
|
|
127
133
|
runs one statement at a time. Merge them instead: CTEs, `UNNEST`, or one multi-row statement.
|
|
128
134
|
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
135
|
+
Write the batch implementation first: it takes an array of keys and returns a keyed map. The
|
|
136
|
+
single-item function is a thin wrapper over it (`addUrl(url)` calls `addUrls([url])`), with no
|
|
137
|
+
parallel per-item SQL path, for lookups and mutations alike. The batch must give a one-item array the
|
|
138
|
+
same atomicity, partial-failure, ordering, duplicate-key, and result-shape semantics as the single
|
|
139
|
+
call; if it doesn't, fix the batch rather than keep a per-item path. Call the batch from loops, `.map`
|
|
140
|
+
callbacks, and `Promise.all` fan-outs. `no-await-in-loop` misses a helper that queries internally
|
|
141
|
+
and `Promise.all(items.map(helper))`; review those call sites by hand.
|
|
133
142
|
|
|
134
143
|
Overlap independent cache and PostgreSQL reads with `Promise.all`, such as a rate-limiter check
|
|
135
|
-
alongside the user fetch. Serialize only true dependencies, where one result feeds the next call.
|
|
144
|
+
alongside the user fetch. Serialize only true dependencies, where one result feeds the next call. Suppress the
|
|
145
|
+
sequential-await lint only when each step consumes the previous step's result; a shared
|
|
146
|
+
transaction client or lock ordering is not a data dependency. A lock that must be taken first
|
|
147
|
+
stays a separate, earlier statement; never reorder or parallelize it. Audit every added suppression.
|
|
148
|
+
|
|
149
|
+
Correctness comes before round trips. Before folding a statement away, state the guarantee it
|
|
150
|
+
gave (what it locked, which snapshot it read) and reject the fold if that weakens. An `EXISTS`
|
|
151
|
+
inside a `FOR UPDATE` statement reads a snapshot taken before the lock wait, so it can be stale.
|
|
136
152
|
|
|
137
153
|
Run a symmetric check, such as "is either party blocking the other", as one query with an `OR`
|
|
138
154
|
or both-direction predicate, not twice with the arguments swapped.
|
|
@@ -4,8 +4,11 @@ UUIDv7 ordering supports range bounds and index-friendly time windows. Generate
|
|
|
4
4
|
instead of extracting timestamps in predicates; use a generated timestamp only when application
|
|
5
5
|
semantics require one. Partition on the range key that queries and retention actually constrain.
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
|
|
7
|
+
Choose a lifecycle. Pre-create future ranges before writes need them, or use on-demand ranges:
|
|
8
|
+
`RANGE (id)` with a DEFAULT partition, split into explicit ranges on demand, and a guarded drop
|
|
9
|
+
that refuses drops inside the retention window or under legal hold. Move or check rows in the
|
|
10
|
+
DEFAULT partition that belong to the range before dropping it. Maintain a default partition
|
|
11
|
+
only with an explicit attachment plan. Before attaching a populated range, prove the default partition excludes that range
|
|
9
12
|
or move conflicting rows; otherwise attachment can scan and lock the default partition.
|
|
10
13
|
|
|
11
14
|
Every primary or unique constraint on a partitioned table must include its partition key. Verify
|
|
@@ -13,6 +13,7 @@ Local policy overrides these portable conventions.
|
|
|
13
13
|
- Joined identities use foreign keys; read [relations](references/relations.md) before modeling references.
|
|
14
14
|
- Split differing shapes, share identical values, and keep history documents as JSON.
|
|
15
15
|
- Use canonical [table shapes](references/table-shapes.md) for revisions, changes, work, and cursors.
|
|
16
|
+
- Default to finite retention for everything except user content; see [table shapes](references/table-shapes.md#retention).
|
|
16
17
|
- Reuse parameterized trigger functions instead of copying bodies.
|
|
17
18
|
- Comment every table, column, and view; document reviewed exceptions explicitly.
|
|
18
19
|
|
|
@@ -2,7 +2,9 @@
|
|
|
2
2
|
|
|
3
3
|
An identity that readers join is an FK column or an FK child row. JSON holds structured,
|
|
4
4
|
schemaless data and history snapshots, including identities recorded as they were at the time.
|
|
5
|
-
Do not use UUID arrays, JSON ids, or type/id pairs for live relations. A
|
|
5
|
+
Do not use UUID arrays, JSON ids, or type/id pairs for live relations. A relation id that
|
|
6
|
+
appears in JSON or in a hashed key also gets an FK column on the same row; the JSON copy may stay,
|
|
7
|
+
and dedup keys on the FK columns, plus any other identity fields, rather than a hash of the document. A column never names a
|
|
6
8
|
table: use one table per target and generate any per-relation progress enum from the same metadata.
|
|
7
9
|
Cursor positions and random fencing tokens are intentionally not live entity references; record
|
|
8
10
|
their reviewed exception instead of inventing a foreign key.
|
|
@@ -7,14 +7,14 @@ explicit sweeper-written timestamp; `now()` cannot appear in an immutable genera
|
|
|
7
7
|
When state keeps returning to earlier states, record append-only attempts or changes and derive
|
|
8
8
|
current state from the latest row rather than erasing the previous lifecycle.
|
|
9
9
|
|
|
10
|
-
| Concept | Shape
|
|
11
|
-
| -------------------------- |
|
|
12
|
-
| Field-diff history | `<entity>_revisions`: UUIDv7 `id`, entity FK, nullable revision actor FK with `ON DELETE SET NULL`, revision-type enum, `changes jsonb`.
|
|
13
|
-
| Typed lifecycle ledger | `<entity>_changes`: UUIDv7 `id`, entity FK, change-type enum, retained actor identity FK with `ON DELETE RESTRICT`, typed joined payload and JSON for unjoined data. Time comes from `id`.
|
|
14
|
-
| Current state from a log | Latest change by `id DESC`, projected through a shared trigger or read through a view.
|
|
15
|
-
| Per-entity work | `<entity>_<job>_work_items`: entity FK, lease and final-outcome columns, claim index, retention.
|
|
16
|
-
| Job progress | `<job>_cursors
|
|
17
|
-
| Inbound or protocol events | Append-only `<source>_events` with `occurred_at` recording the event's business time.
|
|
10
|
+
| Concept | Shape |
|
|
11
|
+
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
12
|
+
| Field-diff history | `<entity>_revisions`: UUIDv7 `id`, entity FK, nullable revision actor FK with `ON DELETE SET NULL`, revision-type enum, `changes jsonb`. |
|
|
13
|
+
| Typed lifecycle ledger | `<entity>_changes`: UUIDv7 `id`, entity FK, change-type enum, retained actor identity FK with `ON DELETE RESTRICT`, typed joined payload and JSON for unjoined data. Time comes from `id`. |
|
|
14
|
+
| Current state from a log | Latest change by `id DESC`, projected through a shared trigger or read through a view. |
|
|
15
|
+
| Per-entity work | Default to a queue job id plus a bounded re-enqueue sweep. When justified, `<entity>_<job>_work_items`: entity FK, lease and final-outcome columns, claim index, retention. |
|
|
16
|
+
| Job progress | Prefer the position in job data. Use `<job>_cursors` only when it must be transactional with domain rows: finite enum key or checked singleton, keyset positions without FKs, optional sweep upper bound. |
|
|
17
|
+
| Inbound or protocol events | Append-only `<source>_events` with `occurred_at` recording the event's business time. |
|
|
18
18
|
|
|
19
19
|
Revisions and changes are append-only through a generic `fn_reject_mutation`, parameterized by
|
|
20
20
|
actor columns that may be erased to NULL. Permit only those erasures; parent-cascade deletion
|
|
@@ -25,20 +25,40 @@ A system change has a NULL actor. Retained actors remain historical identities,
|
|
|
25
25
|
|
|
26
26
|
## Work leases
|
|
27
27
|
|
|
28
|
-
|
|
28
|
+
Default to a queue job whose job id is the durable row id, enqueued after commit, plus a bounded
|
|
29
|
+
sweep that re-enqueues rows with no recorded outcome. The queue already supplies attempts,
|
|
30
|
+
backoff, locking, stall recovery and unique ids. A lease or outbox work-item table needs a stated
|
|
31
|
+
reason: legal or payment records, old values that vanish at commit, or an outcome that must
|
|
32
|
+
commit with domain rows.
|
|
33
|
+
|
|
34
|
+
When a table is justified, fence with one compare-and-set on the outcome or a generation
|
|
35
|
+
(`WHERE completed_at IS NULL` / `AND generation = $n`) instead of a full set of lease columns.
|
|
36
|
+
Idempotent database-only work needs no fence. A unique job id stops duplicate enqueues, not
|
|
37
|
+
duplicate runs.
|
|
38
|
+
|
|
39
|
+
When a full lease is needed, the lease lives on a row that exists only for the work. If an invoice outlives a delivery job,
|
|
29
40
|
use `invoice_delivery_work_items`, with an invoice FK and `ON DELETE CASCADE`, rather than putting
|
|
30
41
|
the lease on the invoice. Do not create a shared polymorphic lease table.
|
|
31
42
|
|
|
32
|
-
|
|
43
|
+
A full lease uses `lease_token uuid`, `leased_at`, `lease_expires_at`, `attempt_count`, `available_at`, and
|
|
33
44
|
`completed_at` plus timestamps for final outcomes. Add `generation` only when re-enqueue can race
|
|
34
45
|
a running worker. Each claim or reclaim gets a fresh opaque token. Every renewal, completion,
|
|
35
46
|
and failure must include `WHERE lease_token = $mine`; zero affected rows means ownership was lost.
|
|
36
47
|
Keep retries in an optional `_attempts` ledger with `attempt_number` when their lifecycle must
|
|
37
48
|
survive. One-shot compare-and-set claims and human assignments need their own shapes.
|
|
38
49
|
|
|
50
|
+
## Retention
|
|
51
|
+
|
|
52
|
+
Give every durable row a finite retention window, except user content. Prefer a compact per-entity
|
|
53
|
+
summary over raw event rows when no reader needs the individual events; append-only ledgers still
|
|
54
|
+
get a retention window. Delete a cache's older rows when a new version is written.
|
|
55
|
+
A bounded post-commit enqueue-loss window is acceptable when the stale result heals itself through
|
|
56
|
+
a TTL; no transactional dirty-work table is needed.
|
|
57
|
+
|
|
39
58
|
## Cursors and shared functions
|
|
40
59
|
|
|
41
|
-
|
|
60
|
+
Prefer keeping the position in job data; a cursor table is for a sweep whose position must commit with
|
|
61
|
+
its domain rows. A cursor key names the finite thing being swept. A singleton uses
|
|
42
62
|
`is_singleton boolean PRIMARY KEY DEFAULT true CHECK (is_singleton)`; the job seeds its own row
|
|
43
63
|
with `ON CONFLICT DO NOTHING`. `cursor_<thing>_id` and optional `sweep_upper_bound_<thing>_id` are
|
|
44
64
|
FK-free UUIDv7 positions; NULL means the start. Per-unit work progress belongs in work items.
|