aspora-threatlenscli 0.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
# ── Python ────────────────────────────────────────────────────────────────────
|
|
2
|
+
__pycache__/
|
|
3
|
+
*.py[cod]
|
|
4
|
+
*.pyo
|
|
5
|
+
*.pyd
|
|
6
|
+
*.egg
|
|
7
|
+
*.egg-info/
|
|
8
|
+
dist/
|
|
9
|
+
build/
|
|
10
|
+
.eggs/
|
|
11
|
+
*.whl
|
|
12
|
+
.venv/
|
|
13
|
+
venv/
|
|
14
|
+
env/
|
|
15
|
+
.env
|
|
16
|
+
.env.*
|
|
17
|
+
!.env.example
|
|
18
|
+
# Claude Code: local agent config, skills and worktrees — never pushed.
|
|
19
|
+
.claude/
|
|
20
|
+
CLAUDE.md
|
|
21
|
+
|
|
22
|
+
# ── SQLite ────────────────────────────────────────────────────────────────────
|
|
23
|
+
*.db
|
|
24
|
+
*.db-shm
|
|
25
|
+
*.db-wal
|
|
26
|
+
*.sqlite
|
|
27
|
+
*.sqlite3
|
|
28
|
+
|
|
29
|
+
# ── Node / Next.js ────────────────────────────────────────────────────────────
|
|
30
|
+
node_modules/
|
|
31
|
+
.next/
|
|
32
|
+
out/
|
|
33
|
+
.vercel/
|
|
34
|
+
npm-debug.log*
|
|
35
|
+
yarn-debug.log*
|
|
36
|
+
yarn-error.log*
|
|
37
|
+
pnpm-debug.log*
|
|
38
|
+
*.tsbuildinfo
|
|
39
|
+
next-env.d.ts
|
|
40
|
+
|
|
41
|
+
# ── Docker ────────────────────────────────────────────────────────────────────
|
|
42
|
+
docker-compose.override.yml
|
|
43
|
+
|
|
44
|
+
# ── Temporal ─────────────────────────────────────────────────────────────────
|
|
45
|
+
temporal/dynamicconfig/*.yaml
|
|
46
|
+
!temporal/dynamicconfig/development.yaml
|
|
47
|
+
|
|
48
|
+
# ── OS / IDE ──────────────────────────────────────────────────────────────────
|
|
49
|
+
.DS_Store
|
|
50
|
+
.DS_Store?
|
|
51
|
+
._*
|
|
52
|
+
.Spotlight-V100
|
|
53
|
+
.Trashes
|
|
54
|
+
Thumbs.db
|
|
55
|
+
.idea/
|
|
56
|
+
.vscode/
|
|
57
|
+
*.swp
|
|
58
|
+
*.swo
|
|
59
|
+
|
|
60
|
+
# ── Logs / coverage ───────────────────────────────────────────────────────────
|
|
61
|
+
*.log
|
|
62
|
+
logs/
|
|
63
|
+
coverage/
|
|
64
|
+
.coverage
|
|
65
|
+
htmlcov/
|
|
66
|
+
.pytest_cache/
|
|
67
|
+
.mypy_cache/
|
|
68
|
+
.ruff_cache/
|
|
69
|
+
|
|
70
|
+
# ── Secrets ───────────────────────────────────────────────────────────────────
|
|
71
|
+
*.pem
|
|
72
|
+
*.key
|
|
73
|
+
*.cert
|
|
74
|
+
secrets/
|
|
75
|
+
# SC-07 / SC-03 — credential material that has shown up in repos of this shape.
|
|
76
|
+
# `~/.threatlenscli/config.json` holds a live API key; the rest are the private-key and
|
|
77
|
+
# keystore formats the .pem/.key patterns above miss.
|
|
78
|
+
.threatlenscli/
|
|
79
|
+
*.p12
|
|
80
|
+
*.pfx
|
|
81
|
+
*.jks
|
|
82
|
+
*.keystore
|
|
83
|
+
*.crt
|
|
84
|
+
id_rsa
|
|
85
|
+
id_rsa.*
|
|
86
|
+
id_ed25519
|
|
87
|
+
id_ed25519.*
|
|
88
|
+
.envrc
|
|
89
|
+
# Never commit a real Fernet key for SECRETS_KEY (see backend/src/app/crypto.py).
|
|
90
|
+
secrets.key
|
|
91
|
+
*.secrets.json
|
|
@@ -0,0 +1,445 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: aspora-threatlenscli
|
|
3
|
+
Version: 0.4.0
|
|
4
|
+
Summary: Image, code and open-source dependency scanning in your CI, reported to a ThreatLens server
|
|
5
|
+
Keywords: ci,container-scanning,gitleaks,sast,sca,security,semgrep,trivy
|
|
6
|
+
Classifier: Development Status :: 4 - Beta
|
|
7
|
+
Classifier: Environment :: Console
|
|
8
|
+
Classifier: Intended Audience :: Developers
|
|
9
|
+
Classifier: Operating System :: MacOS
|
|
10
|
+
Classifier: Operating System :: POSIX
|
|
11
|
+
Classifier: Programming Language :: Python :: 3
|
|
12
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
13
|
+
Classifier: Topic :: Security
|
|
14
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
15
|
+
Requires-Python: >=3.8
|
|
16
|
+
Description-Content-Type: text/markdown
|
|
17
|
+
|
|
18
|
+
# threatlenscli — ThreatLens CLI
|
|
19
|
+
|
|
20
|
+
Snyk-style scanning in **your** CI. The scanners run on the runner; only the
|
|
21
|
+
results go to your ThreatLens server. The server never builds your code, never pulls
|
|
22
|
+
your private images and never needs a Docker socket.
|
|
23
|
+
|
|
24
|
+
| Command | Scans | Engines (on this machine) | Snyk equivalent |
|
|
25
|
+
|---|---|---|---|
|
|
26
|
+
| `threatlenscli oss [PATH]` | open-source dependencies, incl. transitive | `trivy fs` (lockfiles) ∪ `trivy rootfs` (installed artefacts) ∪ optional build-tool SBOM | `snyk test` / `snyk monitor` |
|
|
27
|
+
| `threatlenscli code [PATH]` | SAST, code quality, secrets, IaC misconfig | Semgrep CE, Gitleaks (else Trivy secret), Trivy misconfig | `snyk code test` + `snyk iac test` |
|
|
28
|
+
| `threatlenscli image <REF>` (`scan`, `container`) | container image: OS + library vulns, secrets, misconfig | Trivy (same flags as the server) | `snyk container test` |
|
|
29
|
+
|
|
30
|
+
It is a single Python file with zero pip dependencies (Python 3.8+).
|
|
31
|
+
|
|
32
|
+
## Install
|
|
33
|
+
|
|
34
|
+
From PyPI (the package is `aspora-threatlenscli`; the command is `threatlenscli`):
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
pipx install aspora-threatlenscli # or: pip install aspora-threatlenscli
|
|
38
|
+
pipx install aspora-threatlenscli==0.4.0 # in CI: pin the version
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Or the single file straight from the repository:
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
sudo install -m 755 cli/threatlenscli /usr/local/bin/threatlenscli # or: ./cli/threatlenscli install
|
|
45
|
+
threatlenscli --version # threatlenscli 0.2.0
|
|
46
|
+
threatlenscli doctor # checks everything below
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
In CI, install threatlenscli from the commit you reviewed and verify it (SUP-1/SUP-5):
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
curl -fsSLo threatlenscli https://raw.githubusercontent.com/<org>/<repo>/<40-char-sha>/cli/threatlenscli
|
|
53
|
+
echo "<sha256 from: git show <sha>:cli/threatlenscli | sha256sum> threatlenscli" | sha256sum -c -
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Requirements on the runner:
|
|
57
|
+
|
|
58
|
+
- **Trivy >= 0.53** on `PATH`. Install a pinned release and verify its checksum.
|
|
59
|
+
threatlenscli refuses the credential-stealing releases **0.69.4, 0.69.5 and 0.69.6**
|
|
60
|
+
(GHSA-69fq-xp46-6x23). If `doctor` ever reports one, remove it and rotate every
|
|
61
|
+
secret that runner can reach. Add more entries with
|
|
62
|
+
`THREATLENS_SCANNER_VERSION_DENYLIST=trivy:x.y.z,...`.
|
|
63
|
+
- For `code`: **semgrep**, installed hash-locked from the server's
|
|
64
|
+
`backend/semgrep.lock` of the reviewed commit (`python3.14 -m venv /opt/semgrep &&
|
|
65
|
+
/opt/semgrep/bin/pip install --require-hashes --only-binary=:all: -r backend/semgrep.lock`;
|
|
66
|
+
the lock is resolved for CPython 3.14 on Linux, so another interpreter fails
|
|
67
|
+
loudly instead of installing unhashed files) and, optionally, **gitleaks** (v8
|
|
68
|
+
with the `dir` command). Without semgrep the code scan has no SAST and no
|
|
69
|
+
quality rules. threatlenscli says so on stderr, and `--require-sast` turns that into
|
|
70
|
+
exit 2. semgrep and gitleaks are checked against the same denylist
|
|
71
|
+
(`THREATLENS_SCANNER_VERSION_DENYLIST=semgrep:x.y.z,gitleaks:x.y.z`); an unparseable
|
|
72
|
+
version exits 2 (fail closed).
|
|
73
|
+
- For `image`: the image must be reachable by Trivy (local Docker daemon or a
|
|
74
|
+
registry). Registry credentials go in `TRIVY_USERNAME` / `TRIVY_PASSWORD`, never on argv.
|
|
75
|
+
|
|
76
|
+
GitHub Actions: pin every third-party action (including any Trivy action) by
|
|
77
|
+
its full commit SHA. The trivy-action and setup-trivy tags were compromised in
|
|
78
|
+
the same incident.
|
|
79
|
+
|
|
80
|
+
## Authenticate
|
|
81
|
+
|
|
82
|
+
An admin creates a key in **Settings → API Keys**. Give CI keys the narrowest scope:
|
|
83
|
+
|
|
84
|
+
| Command | Scope | Legacy scope (still accepted, audited as deprecated) |
|
|
85
|
+
|---|---|---|
|
|
86
|
+
| `threatlenscli oss` | `ingest:oss` | `code:write` |
|
|
87
|
+
| `threatlenscli code` | `ingest:code` | `code:write` |
|
|
88
|
+
| `threatlenscli image` | `ingest:image` | `images:write` |
|
|
89
|
+
|
|
90
|
+
An `ingest:*` key can upload results and read back its own runs. It cannot read
|
|
91
|
+
other findings, add images or trigger server scans. Keys expire after 90 days
|
|
92
|
+
by default.
|
|
93
|
+
|
|
94
|
+
**Allowed identities.** A key can be bound to the projects it may write, using
|
|
95
|
+
glob patterns over normalised identities: `github.com/acme/*` or
|
|
96
|
+
`github.com/acme/payments` (`*` matches one path segment, `**` any number).
|
|
97
|
+
With an empty list, the key may only write projects it created itself. An
|
|
98
|
+
upload for any other project gets 403, and the refusal is audited. Set these
|
|
99
|
+
bindings for every shared CI key, and again after rotating a key.
|
|
100
|
+
|
|
101
|
+
```bash
|
|
102
|
+
threatlenscli login # asks for your ThreatLens URL, opens the browser: sign in, approve, done
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
`login` opens the ThreatLens sign-in page in your browser (or prints the link, for
|
|
106
|
+
SSH sessions). Sign in, check that the code matches your terminal, and approve; the
|
|
107
|
+
terminal then says `Authenticated as you@company.com` and every command works. The
|
|
108
|
+
CLI receives its own upload-only key (90 days, revocable; `threatlenscli logout`
|
|
109
|
+
revokes it). Admins' CLIs may upload for any repository; a member's CLI only for
|
|
110
|
+
projects and branches it creates itself. To paste a key instead:
|
|
111
|
+
`threatlenscli login --with-api-key`.
|
|
112
|
+
|
|
113
|
+
No server is built in. In CI set both environment variables; they always win over
|
|
114
|
+
`~/.threatlenscli/config.json`. On a CI runner an env key is never paired with the
|
|
115
|
+
server a stale config on a shared runner names:
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
export THREATLENS_SERVER=${{ vars.THREATLENS_SERVER }} # e.g. https://threatlens.example.com
|
|
119
|
+
export THREATLENS_API_KEY=${{ secrets.THREATLENS_API_KEY }}
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
|
|
123
|
+
Transport rules:
|
|
124
|
+
- TLS certificates are always verified. Use `SSL_CERT_FILE` for a private CA.
|
|
125
|
+
- Plain `http://` works only for loopback. For another host, outside CI only,
|
|
126
|
+
pass `login --insecure` or set `THREATLENS_ALLOW_INSECURE=1`. It is never allowed in CI.
|
|
127
|
+
- Redirects to another host or port, or from https to http, are refused, so the
|
|
128
|
+
key is never forwarded. A key saved for server A is never sent to a different
|
|
129
|
+
`THREATLENS_SERVER`.
|
|
130
|
+
- Reads (GET) retry 429, 502, 503, 504 and network errors with backoff
|
|
131
|
+
(`Retry-After` honoured). The upload (POST) is retried only when the server
|
|
132
|
+
explicitly did not process it (429, or 503 `ingest_busy`/`scan_in_progress`)
|
|
133
|
+
or the request never left the runner (connection refused, DNS, send timeout).
|
|
134
|
+
A lost response (read timeout, reset, 502/504) is NOT retried: the server may
|
|
135
|
+
have stored the results, so the CLI exits 2 and says to check the UI.
|
|
136
|
+
`--upload-timeout` (default 300 s) sets how long to wait for the response.
|
|
137
|
+
Large uploads are gzip-compressed.
|
|
138
|
+
|
|
139
|
+
## Exit codes (all scan commands)
|
|
140
|
+
|
|
141
|
+
| Code | Meaning |
|
|
142
|
+
|---|---|
|
|
143
|
+
| `0` | scan completed, nothing at or above the gate |
|
|
144
|
+
| `1` | scan completed, gating issues found |
|
|
145
|
+
| `2` | error: scanner failure or timeout, network/TLS, HTTP 4xx/5xx, upload refused, bad flag (`--fail-on patchable`, unknown `--fail-on` value), `--require-sast` without semgrep, denylisted or too-old Trivy |
|
|
146
|
+
| `3` | nothing scannable. `oss`: no package resolved by any pass (the unresolved manifests are listed). `code`: no file to scan. Never for images |
|
|
147
|
+
|
|
148
|
+
## Gating (CI policy)
|
|
149
|
+
|
|
150
|
+
```
|
|
151
|
+
--severity-threshold low|medium|high|critical issues at or ABOVE this severity gate (default low;
|
|
152
|
+
'unknown' gates as medium)
|
|
153
|
+
--fail-on all fail only if a gating issue has a fix
|
|
154
|
+
--fail-on upgradable fail only if upgrading fixes it (unknown upgradability + a fix counts as upgradable)
|
|
155
|
+
--fail-on none report, never fail
|
|
156
|
+
(`threatlenscli code` ignores all/upgradable with a warning: secrets and SAST
|
|
157
|
+
findings have no upgrade fix, so those values would disable the gate)
|
|
158
|
+
--include-dev include dev dependencies (npm, yarn, gradle) — off by default, like `snyk test`.
|
|
159
|
+
Without it, dev dependencies are removed from every pass, including the copies
|
|
160
|
+
`npm ci` installed in node_modules (matched by purl, else name+version) and
|
|
161
|
+
binaries shipped inside a dev-only package (esbuild's Go binary), so they
|
|
162
|
+
neither gate nor upload; the count is printed and recorded in meta/coverage
|
|
163
|
+
--exclude-base-image-vulns image only: ignore OS vulns that come from the base image (needs --file)
|
|
164
|
+
--require-fresh-db exit 2 unless the Trivy DB was built within 48 h and is not past its NextUpdate
|
|
165
|
+
(useful with --offline runners that restore a cached DB)
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
Code-quality and licence findings are reported (JSON, SARIF, the UI) but never fail
|
|
169
|
+
the build; `threatlenscli code --gate-quality` lets them gate too.
|
|
170
|
+
|
|
171
|
+
The gate uses the **server's triage**: findings marked false positive, accepted
|
|
172
|
+
risk or not affected (and not past their `suppress_until`) do not fail the build.
|
|
173
|
+
Nothing else carries over. A finding this scan reports is present, so a
|
|
174
|
+
default-branch row that the pipeline closed (`resolved`, `fixed`, `stale`,
|
|
175
|
+
`not_applicable`) still gates: a PR that brings back a vulnerability `main`
|
|
176
|
+
already fixed fails. Only an advisory withdrawn upstream (`rejected`) is exempt.
|
|
177
|
+
With `--no-upload` the gate runs on local results only, without triage; threatlenscli
|
|
178
|
+
notes this on stderr.
|
|
179
|
+
|
|
180
|
+
`--fail-on critical,high` (the old severity list) is still accepted. It now means
|
|
181
|
+
`--severity-threshold high`: at or above, so a critical fails a "high" gate.
|
|
182
|
+
|
|
183
|
+
## Outputs
|
|
184
|
+
|
|
185
|
+
| Flag | Output |
|
|
186
|
+
|---|---|
|
|
187
|
+
| `--json` | exactly one JSON document on stdout (`schema: vss-cli-result/1`); all progress goes to stderr |
|
|
188
|
+
| `--json-file-output PATH` | the same document, written to a file |
|
|
189
|
+
| `--sarif-file-output PATH` | SARIF 2.1.0 for GitHub code scanning: one run per engine with distinct categories (`vss/oss/`, `vss/image/`, `vss/code/semgrep/`, `vss/code/gitleaks/`, `vss/code/trivy/`), text only, repo-relative paths, redacted |
|
|
190
|
+
| `--sbom-file-output PATH` | CycloneDX from `trivy convert` (inventory only, no vulnerabilities). Trivy 0.74 writes specVersion 1.7. For `oss` it covers every pass (lockfiles, installed artefacts, `--sbom`) after dev dependencies are excluded (unless `--include-dev`) |
|
|
191
|
+
|
|
192
|
+
JSON and SARIF are redacted: secrets never appear, even for public repositories
|
|
193
|
+
whose CI artefacts are public. No runner path leaves the machine: the report
|
|
194
|
+
`ArtifactName` and the SBOM `metadata.component.name` carry the project identity,
|
|
195
|
+
and absolute paths (target, `$VIRTUAL_ENV`, temp dir, `--artifact`, `--sbom`)
|
|
196
|
+
are replaced by placeholders in uploads and output files.
|
|
197
|
+
|
|
198
|
+
Trivy DB (C14): unless `--offline`, every scan first runs `--download-db-only`,
|
|
199
|
+
then `--download-java-db-only` (oss/image), then — for image and code — a
|
|
200
|
+
misconfig checks-bundle prefetch (a `trivy config` of an empty private dir).
|
|
201
|
+
Sources are pinned to `mirror.gcr.io` / `ghcr.io` unless the runner sets its own
|
|
202
|
+
`TRIVY_DB_REPOSITORY` / `TRIVY_JAVA_DB_REPOSITORY` / `TRIVY_CHECKS_BUNDLE_REPOSITORY`.
|
|
203
|
+
Every scan then runs with `--skip-db-update --skip-java-db-update
|
|
204
|
+
--skip-check-update`. A failed prefetch is a warning (the embedded checks are
|
|
205
|
+
used) and is recorded as `meta.check_bundle_prefetch` / `check_bundle_stale`.
|
|
206
|
+
|
|
207
|
+
## `threatlenscli oss` — run it after the build
|
|
208
|
+
|
|
209
|
+
```bash
|
|
210
|
+
npm ci # or: pip install -r requirements.txt into a venv / mvn package / gradle build
|
|
211
|
+
threatlenscli oss . --severity-threshold high --sarif-file-output oss.sarif
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
- Pass 1, `trivy fs`: reads lockfiles with the dependency graph (relationship,
|
|
215
|
+
`introduced_through`, dev flag, licences). It runs in `precise` mode, so an
|
|
216
|
+
unpinned manifest yields no package rather than a made-up version.
|
|
217
|
+
- Pass 2, `trivy rootfs`: reads what is installed (`node_modules`,
|
|
218
|
+
`site-packages`, a venv (`$VIRTUAL_ENV` is added automatically), JAR/WAR
|
|
219
|
+
files, Go and Rust binaries). `--artifact PATH` adds built outputs;
|
|
220
|
+
`--lockfile-only` skips this pass.
|
|
221
|
+
- JVM builds keep dependencies in `~/.m2` / `~/.gradle`, outside the project:
|
|
222
|
+
- `--sbom target/bom.json` ingests a cyclonedx-maven-plugin or
|
|
223
|
+
cyclonedx-gradle-plugin SBOM (re-matched by `trivy sbom`, which takes no
|
|
224
|
+
`--skip-check-update`: it has no misconfig checks). The path is checked
|
|
225
|
+
(exists, <= 128 MiB, CycloneDX or SPDX JSON) before any Trivy pass runs.
|
|
226
|
+
- Manifests without a lockfile are resolved automatically **on a CI runner**,
|
|
227
|
+
like `snyk test` (elsewhere pass `--resolve`; `--no-resolve` turns it off):
|
|
228
|
+
1. a workspace lockfile in a parent folder (npm/pnpm/yarn, uv/poetry/pdm, Cargo);
|
|
229
|
+
2. the ecosystem's own tool, writing into a private dir, never the repo:
|
|
230
|
+
`npm install --package-lock-only --ignore-scripts`, `pip install --dry-run
|
|
231
|
+
--report`, `composer update --no-install --no-scripts --no-plugins`,
|
|
232
|
+
`bundle lock`, `cargo generate-lockfile`, `dotnet restore --use-lock-file`,
|
|
233
|
+
`mvn dependency:copy-dependencies`, a Gradle init-script task. The nearest
|
|
234
|
+
`gradlew`/`mvnw` up to the git root is used (multi-module builds);
|
|
235
|
+
3. JVM only: the JARs built in a parent build root (Dockerfile in
|
|
236
|
+
`services/api`, `./gradlew assemble` at the repo root).
|
|
237
|
+
A missing tool or a failed run is a warning, never fatal. Build tools
|
|
238
|
+
execute your build configuration; the server never does.
|
|
239
|
+
- Detected manifests that did not resolve (`build.gradle` without
|
|
240
|
+
`gradle.lockfile`, unpinned `requirements.txt`, `.csproj` without
|
|
241
|
+
`packages.lock.json`, `requirements.lock`, ...) are listed with a fix. When no
|
|
242
|
+
package resolves at all, the exit code is 3, never a false "no
|
|
243
|
+
vulnerabilities".
|
|
244
|
+
- Identity is the normalised git remote (`github.com/acme/api`, credentials
|
|
245
|
+
stripped) plus the sub-path in a monorepo. Without a remote, pass
|
|
246
|
+
`--project-name`. CLI projects are separate from GitHub-imported ones; the UI
|
|
247
|
+
links them by remote.
|
|
248
|
+
- Only the **default branch** is monitored: its upload becomes the project's
|
|
249
|
+
inventory, findings are reconciled, the daily re-check watches it and Slack
|
|
250
|
+
alerts fire. Uploads from other branches (PRs, feature branches) are stored
|
|
251
|
+
as separate branch snapshots. They return triage-aware gate results and never
|
|
252
|
+
touch default-branch findings.
|
|
253
|
+
- The default branch comes from `refs/remotes/origin/HEAD`, `CI_DEFAULT_BRANCH`
|
|
254
|
+
(GitLab), `BUILDKITE_PIPELINE_DEFAULT_BRANCH`, or on GitHub Actions (whose
|
|
255
|
+
shallow checkout has no origin/HEAD) `repository.default_branch` in
|
|
256
|
+
`$GITHUB_EVENT_PATH`. Once the server has stored a project's default branch it
|
|
257
|
+
keeps it: a different claim from a client is logged, audited and ignored.
|
|
258
|
+
With no default known at all, only `main`/`master` are monitored, and that
|
|
259
|
+
guess is not stored.
|
|
260
|
+
- A pull/merge-request build is **never** monitored, whatever its head branch
|
|
261
|
+
is called (a fork PR from its own `main`). threatlenscli detects it from
|
|
262
|
+
`GITHUB_HEAD_REF`/`GITHUB_EVENT_NAME`, `CI_MERGE_REQUEST_IID`,
|
|
263
|
+
`BITBUCKET_PR_ID`, `BUILDKITE_PULL_REQUEST`, `CIRCLE_PULL_REQUEST`,
|
|
264
|
+
`SYSTEM_PULLREQUEST_PULLREQUESTID` or `CHANGE_ID`.
|
|
265
|
+
- The server re-derives everything from the raw Trivy reports (it never trusts
|
|
266
|
+
the client's merge) and unions them with a live OSV query of every versioned
|
|
267
|
+
package, so a finding carries `engines: trivy`, `osv` or both. An OSV outage
|
|
268
|
+
never resolves OSV-found findings.
|
|
269
|
+
- An upload resolves findings that disappeared only when it is provably
|
|
270
|
+
complete: fresh Trivy DB (not `--offline`), all requested passes present, no
|
|
271
|
+
package-count collapse, and no manifest that resolved last run and is
|
|
272
|
+
unresolved now (a monorepo sub-project whose install failed). Otherwise
|
|
273
|
+
nothing is resolved and the run's coverage says why.
|
|
274
|
+
|
|
275
|
+
## `threatlenscli code`
|
|
276
|
+
|
|
277
|
+
```bash
|
|
278
|
+
threatlenscli code . --require-sast --severity-threshold high --sarif-file-output code.sarif
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
- **Semgrep** packs: the security base (`p/default`, `p/owasp-top-ten`,
|
|
282
|
+
`p/cwe-top-25`, `p/secrets`, `p/security-audit`), language and framework
|
|
283
|
+
packs from detected files, IaC/CI packs, and quality packs (`p/r2c-bug-scan`,
|
|
284
|
+
`p/r2c-best-practices`, `r/<lang>.lang.correctness`, ...). The server's list
|
|
285
|
+
(`GET /api/code/config?languages=…&frameworks=…&iac=…`, sent the local
|
|
286
|
+
detection) wins when it is available; `coverage.semgrep_pack_source` says which
|
|
287
|
+
list ran. The local fallback (`--no-upload`, or the server unreachable) uses
|
|
288
|
+
tables that mirror the server's default `select_configs` exactly (a parity
|
|
289
|
+
test compares them), including `p/ci` for CI configs (`.github/workflows`,
|
|
290
|
+
`.gitlab-ci.yml`, `Jenkinsfile`, ...); a server-side `CODE_SEMGREP_CONFIGS`
|
|
291
|
+
override is not applied locally (`coverage.semgrep_pack_note`). Semgrep always
|
|
292
|
+
runs with `--metrics=off`, never `--config auto`, `--disable-nosem` and
|
|
293
|
+
`--verbose` (only then does Semgrep report `paths.skipped`; its log goes to a
|
|
294
|
+
file in the private dir). Files over `--semgrep-max-target-bytes` (default
|
|
295
|
+
1000000, the server's) or that time out / fail to parse are counted in
|
|
296
|
+
`coverage.semgrep` (`skipped_paths`, `skipped_by_reason`), warned about, and
|
|
297
|
+
uploaded as repo-relative `paths.skipped` so the server never resolves a
|
|
298
|
+
finding in a file Semgrep did not read. Extra configs:
|
|
299
|
+
`--semgrep-config p/jwt` or a local rules file.
|
|
300
|
+
- **Gitleaks** (when installed) runs without `--redact` into a 0600 report
|
|
301
|
+
inside a private temp dir. threatlenscli digests each secret (sha256), then drops
|
|
302
|
+
the raw value and masks Match/Line before anything is uploaded; the report is
|
|
303
|
+
deleted. gitleaks always honours a repo `.gitleaksignore`, and threatlenscli reports
|
|
304
|
+
when one exists. Without gitleaks, Trivy's secret scanner is used, and the
|
|
305
|
+
digest is recovered from the file, not from Trivy's masked match. Gitleaks
|
|
306
|
+
skips files over `--gitleaks-max-target-mb` (default 50, the server's
|
|
307
|
+
`GITLEAKS_MAX_TARGET_MB`) without a word, so threatlenscli lists them itself:
|
|
308
|
+
`coverage.bounds.GITLEAKS_MAX_TARGET_MB` (`value`, `hit`, `skipped_sample`),
|
|
309
|
+
`coverage.gitleaks_skipped.paths` and a warning.
|
|
310
|
+
- **Trivy** misconfig: Dockerfile, Kubernetes, Helm, CloudFormation, Azure ARM,
|
|
311
|
+
Terraform plan JSON and Ansible (`--misconfig-scanners
|
|
312
|
+
azure-arm,cloudformation,dockerfile,helm,kubernetes,terraformplan-json,ansible`),
|
|
313
|
+
with `**/<dir>` skip globs. **Trivy's Terraform scanner never runs in threatlenscli**:
|
|
314
|
+
it resolves `module` sources from the scanned tree, including loopback
|
|
315
|
+
addresses no proxy can block, so a fork PR's `.tf` could make the runner
|
|
316
|
+
connect where it chooses. When `.tf`, `.tf.json`, `.tofu` or `.tofu.json` files
|
|
317
|
+
are present, threatlenscli warns and records `coverage.terraform_not_scanned`; only
|
|
318
|
+
Semgrep `p/terraform` rules cover them. Full Terraform IaC coverage needs the
|
|
319
|
+
server code scan (which scans a guarded copy of the tree).
|
|
320
|
+
Trivy runs with **no network**: a dead loopback proxy (`HTTPS_PROXY`/`HTTP_PROXY`/
|
|
321
|
+
`ALL_PROXY` in both cases, empty `NO_PROXY`/`no_proxy`), `GIT_ALLOW_PROTOCOL=vss-none`
|
|
322
|
+
and a private `TMPDIR`.
|
|
323
|
+
- Semgrep's secret digest is sha256 of the credential literal (one quoted
|
|
324
|
+
literal, or the value of a single `NAME=VALUE` assignment), the same rule as
|
|
325
|
+
the server and Gitleaks/Trivy, so one secret found by several engines is one finding.
|
|
326
|
+
- `--tracked-only` reports only files git tracks (`git ls-files -z`, so
|
|
327
|
+
non-ASCII names match). Committed `.env` and `.github/**` files are kept. If
|
|
328
|
+
`git ls-files` fails or times out, `--tracked-only` exits 2 instead of
|
|
329
|
+
filtering against an unknown list.
|
|
330
|
+
- Secret findings from Semgrep are masked before upload, in `--json` and in
|
|
331
|
+
SARIF: the matched span, each of its lines, quoted literals inside it and
|
|
332
|
+
every metavariable value (what rule messages interpolate) are masked on the
|
|
333
|
+
full source line before it is clipped. If a literal would still be visible,
|
|
334
|
+
the snippet is dropped and the message becomes the rule title. The mask is the
|
|
335
|
+
server's: a known token prefix plus the last 2 characters for tokens of 16+
|
|
336
|
+
characters (`ghp_****…i2`), `****…xy` for 24+ characters, and nothing at all
|
|
337
|
+
for shorter secrets (`********`).
|
|
338
|
+
- Local findings (`--no-upload` gate, `--json`, SARIF) use the server's
|
|
339
|
+
identities: one secret found by several engines is ONE finding (merged on file
|
|
340
|
+
+ secret digest, `engines` lists them); misconfig identity is rule + resource +
|
|
341
|
+
a hash of the cause lines + an occurrence index, never a line number, so SARIF
|
|
342
|
+
`partialFingerprints` survive edits above the finding. A secret's local
|
|
343
|
+
fingerprint is keyed with a per-run random key and SARIF omits
|
|
344
|
+
`partialFingerprints` for secrets, so a published artifact carries nothing
|
|
345
|
+
derived from a credential. SARIF tags quality results `correctness`/
|
|
346
|
+
`maintainability`/`performance` and licence results `license` (no `security`
|
|
347
|
+
tag or `security-severity`).
|
|
348
|
+
- `--offline`: Semgrep registry packs (`p/...`, `r/...`) are downloads from
|
|
349
|
+
semgrep.dev, so they are skipped. Only `--semgrep-config <local rules>` run;
|
|
350
|
+
with none, Semgrep is skipped with a warning (`--require-sast` then exits 2).
|
|
351
|
+
No checks-bundle prefetch runs (cached or embedded misconfig checks,
|
|
352
|
+
recorded as `check_bundle_stale`).
|
|
353
|
+
|
|
354
|
+
## `threatlenscli image`
|
|
355
|
+
|
|
356
|
+
```bash
|
|
357
|
+
threatlenscli image my-app:1.4.2 --file Dockerfile --platform linux/amd64 --severity-threshold critical
|
|
358
|
+
```
|
|
359
|
+
|
|
360
|
+
This uses the server's exact Trivy argv, flag for flag and in order
|
|
361
|
+
(`GET /api/images/scan-profile` shows it): `--scanners vuln,secret,misconfig`,
|
|
362
|
+
`--image-config-scanners misconfig,secret`, `--pkg-types os,library`,
|
|
363
|
+
`--list-all-pkgs`, `--skip-check-update`, `--no-progress`,
|
|
364
|
+
`--misconfig-scanners dockerfile,kubernetes,helm,cloudformation,azure-arm,ansible`
|
|
365
|
+
(never Terraform: its module resolution would make the runner fetch
|
|
366
|
+
attacker-chosen URLs), `--detection-priority comprehensive`,
|
|
367
|
+
`--max-image-size 16384MiB` and `--image-src docker,remote -- <ref>`. The only
|
|
368
|
+
differences are no `--cache-dir` and `--offline-scan` with `--offline`. A Trivy
|
|
369
|
+
too old for `--max-image-size` drops it with a warning (`meta.flags_dropped`).
|
|
370
|
+
The scan runs with `GIT_ALLOW_PROTOCOL=vss-none` and a private `TMPDIR`
|
|
371
|
+
(registry egress stays: the image must be pulled). `--file` uploads the
|
|
372
|
+
Dockerfile's FROM lines for base-image attribution and advice.
|
|
373
|
+
|
|
374
|
+
## CI recipes
|
|
375
|
+
|
|
376
|
+
```yaml
|
|
377
|
+
# GitHub Actions — PRs gate, main monitors (same command; the branch decides)
|
|
378
|
+
- run: npm ci
|
|
379
|
+
- run: threatlenscli oss . --severity-threshold high --sarif-file-output oss.sarif
|
|
380
|
+
env: { THREATLENS_SERVER: ${{ vars.THREATLENS_SERVER }}, THREATLENS_API_KEY: ${{ secrets.THREATLENS_OSS_KEY }} }
|
|
381
|
+
- run: threatlenscli code . --require-sast --severity-threshold high --sarif-file-output code.sarif
|
|
382
|
+
env: { THREATLENS_SERVER: ${{ vars.THREATLENS_SERVER }}, THREATLENS_API_KEY: ${{ secrets.THREATLENS_CODE_KEY }} }
|
|
383
|
+
- uses: github/codeql-action/upload-sarif@<full-commit-sha>
|
|
384
|
+
if: always()
|
|
385
|
+
with: { sarif_file: oss.sarif, category: threatlens-oss }
|
|
386
|
+
```
|
|
387
|
+
|
|
388
|
+
```yaml
|
|
389
|
+
# GitLab CI
|
|
390
|
+
threatlens:
|
|
391
|
+
script:
|
|
392
|
+
- threatlenscli oss . --severity-threshold high --json-file-output threatlens-oss.json
|
|
393
|
+
artifacts: { paths: [threatlens-oss.json], when: always }
|
|
394
|
+
```
|
|
395
|
+
|
|
396
|
+
Every upload carries provenance in `meta`: the Trivy, Semgrep and Gitleaks
|
|
397
|
+
versions, the Trivy binary sha256, the Trivy DB `UpdatedAt`, sanitised argv
|
|
398
|
+
(the target and temp dirs become `<target>`/`<tmp>`, any other absolute path keeps
|
|
399
|
+
only its basename, credential flag values are masked), git remote, branch and commit, and the scanner
|
|
400
|
+
environment variables that were ignored (for example a `TRIVY_SEVERITY` set in
|
|
401
|
+
CI).
|
|
402
|
+
|
|
403
|
+
## Other commands
|
|
404
|
+
|
|
405
|
+
```
|
|
406
|
+
threatlenscli login [--server URL] browser sign-in (--with-api-key: paste a key); 0600 config
|
|
407
|
+
threatlenscli logout remove saved config
|
|
408
|
+
threatlenscli config show config (key masked; env overrides flagged)
|
|
409
|
+
threatlenscli doctor scanners, denylist, DB age, server reachability, key scopes + identities
|
|
410
|
+
threatlenscli list [--type repos] images (default) or repositories on the server
|
|
411
|
+
threatlenscli install [--dest DIR] symlink this script onto PATH
|
|
412
|
+
```
|
|
413
|
+
|
|
414
|
+
## Measuring parity with Snyk
|
|
415
|
+
|
|
416
|
+
The JSON document written by `--json-file-output` (`schema: vss-cli-result/1`)
|
|
417
|
+
is the ThreatLens input of the parity harness in `backend/tests/parity/`. Run the scan
|
|
418
|
+
with `--no-upload` on the same machine and day as the Snyk run, for example:
|
|
419
|
+
|
|
420
|
+
```bash
|
|
421
|
+
threatlenscli image nginx:1.25@sha256:<digest> --no-upload --json-file-output threatlens-image.json
|
|
422
|
+
snyk container test nginx:1.25@sha256:<digest> --json-file-output=snyk-container.json
|
|
423
|
+
python3 backend/tests/parity/parity_harness.py compare --snyk-container snyk-container.json --vss threatlens-image.json
|
|
424
|
+
```
|
|
425
|
+
|
|
426
|
+
The corpus, the commands for every target, the match rules and the pass bar
|
|
427
|
+
(recall >= 80 % per product, precision reported) are in
|
|
428
|
+
[`docs/parity-benchmark.md`](../docs/parity-benchmark.md). No parity figure is
|
|
429
|
+
claimed until it has been measured that way.
|
|
430
|
+
|
|
431
|
+
## Honest limitations
|
|
432
|
+
|
|
433
|
+
- Semgrep CE is intraprocedural: there is no cross-function or cross-file taint
|
|
434
|
+
(Snyk Code has it). Pro-only rules are unavailable.
|
|
435
|
+
- Registry rules are under the Semgrep Rules License and are fetched live;
|
|
436
|
+
threatlenscli bundles none.
|
|
437
|
+
- `trivy rootfs` finds installed packages but has no dependency graph, so those
|
|
438
|
+
findings have `relationship: unknown`. Rust binaries are found only when built
|
|
439
|
+
with cargo-auditable.
|
|
440
|
+
- `--fail-on upgradable` knows upgradability exactly only for direct
|
|
441
|
+
dependencies. For transitive ones, "has a fix" is treated as upgradable
|
|
442
|
+
(fails closed).
|
|
443
|
+
- The Trivy DB is the newest build available when the run starts, which can
|
|
444
|
+
lag upstream by up to about 24 h.
|
|
445
|
+
- A code scan does not scan git history for secrets.
|