mapterrain 0.1.2 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +129 -408
- package/SECURITY.md +1 -1
- package/bin/postinstall.js +33 -3
- package/bin/terrain-installer.js +280 -34
- package/package.json +7 -6
package/README.md
CHANGED
|
@@ -1,482 +1,203 @@
|
|
|
1
1
|
# Terrain
|
|
2
2
|
|
|
3
|
-
**
|
|
3
|
+
**Open-source CI pre-flight layer for AI/ML systems and the tests around them. Runs locally. No API key required.**
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
# Homebrew
|
|
7
|
-
brew install pmclSF/terrain/mapterrain
|
|
8
|
-
|
|
9
|
-
# npm
|
|
10
|
-
npm install -g mapterrain
|
|
11
|
-
|
|
12
|
-
cd your-repo
|
|
13
|
-
terrain analyze
|
|
14
|
-
```
|
|
15
|
-
|
|
16
|
-
That's it. No config, no setup, no test execution required.
|
|
5
|
+
Rename `customer_name` to `full_name`, and a prompt in another part of the repo may still ask the model for `customer_name`. Change a retriever, and the eval that would catch the quality drop may never run. Terrain catches those gaps in the PR, before the change merges.
|
|
17
6
|
|
|
18
|
-
|
|
7
|
+
Terrain does that by connecting source code, tests, prompts, schemas, eval outputs, runtime artifacts, and coverage reports into one CI-ready protection graph. It turns that graph into ranked findings, explainable impact, JUnit test cases, GitHub annotations, SARIF for security findings, MCP context for AI assistants, and portfolio rollups across repos.
|
|
19
8
|
|
|
20
|
-
|
|
9
|
+
## Why install Terrain?
|
|
21
10
|
|
|
22
|
-
|
|
11
|
+
Most teams already have tests, coverage, and AI evals. The hard part is knowing whether those signals still protect the change in front of you. A developer can see that unit tests passed or that an eval failed, but still not know whether a PR touched an AI/ML surface, which validations should run, what coverage is missing, or why a downstream regression is connected to their diff.
|
|
23
12
|
|
|
24
|
-
|
|
13
|
+
Terrain gives every PR a pre-merge protection map. In one local command it answers:
|
|
25
14
|
|
|
26
|
-
|
|
15
|
+
- What is weak or missing in the current test system?
|
|
16
|
+
- What does this PR put at risk?
|
|
17
|
+
- Which tests or evals matter for this change?
|
|
18
|
+
- Why did the gate fail, and what should be fixed first?
|
|
27
19
|
|
|
28
|
-
|
|
20
|
+
That makes day-to-day work less manual:
|
|
29
21
|
|
|
30
|
-
-
|
|
31
|
-
-
|
|
32
|
-
-
|
|
33
|
-
-
|
|
34
|
-
- When you change `auth/session.ts`, which 41 of your 18,000 tests actually matter?
|
|
22
|
+
- Instead of tracing code -> API -> prompt -> eval by hand, run `terrain report pr` and see what the change affects.
|
|
23
|
+
- Instead of rerunning every eval because no one knows which one matters, run the validations Terrain selects for the diff.
|
|
24
|
+
- Instead of reading a failed eval in isolation, use the cause path to see which code, schema, prompt, or retrieval change led there.
|
|
25
|
+
- Instead of letting weak assertions, skipped tests, and uncovered exports accumulate quietly, surface them as concrete findings before release.
|
|
35
26
|
|
|
36
|
-
|
|
27
|
+
The result is faster review, fewer broad reruns, clearer fixes, and fewer regressions that only make sense after someone reconstructs the system by hand.
|
|
37
28
|
|
|
38
|
-
|
|
29
|
+
Bring your own stack. Terrain reads what teams already use: `pytest`, `jest`, `go test`, Playwright, Promptfoo, DeepEval, Ragas, Great Expectations, Gauntlet-style eval-result JSON, JUnit, LCOV, Istanbul, and repository metadata. It does not replace those tools; it unifies their evidence into one local, deterministic CI gate.
|
|
39
30
|
|
|
40
|
-
|
|
31
|
+
Project status: Terrain is pre-1.0 and actively developed. In 0.3.0, the stable path is the CLI, local/CI artifact contract, GitHub Actions flow, AI/eval artifact ingestion, and portfolio aggregation. The VS Code extension is alpha; marketplace listings, full LSP integration, the plugin runtime, and some preview rules are future work.
|
|
41
32
|
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
**"We changed auth code — what tests should we worry about?"** Terrain traces your change through the import graph and structural dependencies, returning the impacted tests ranked by confidence, with reason chains explaining each selection.
|
|
45
|
-
|
|
46
|
-
**"A tool flagged something but won't explain why."** Every Terrain finding carries an evidence chain. `terrain explain` shows exactly what signals, dependency paths, and scoring rules produced each decision.
|
|
47
|
-
|
|
48
|
-
**"We're migrating frameworks and need to know what's blocking us."** Migration readiness, blockers by type, and preview-scoped difficulty assessment — all derived from static analysis.
|
|
49
|
-
|
|
50
|
-
## The Four Canonical Workflows
|
|
51
|
-
|
|
52
|
-
Terrain is organized around four questions. Everything else is a supporting view.
|
|
53
|
-
|
|
54
|
-
```
|
|
55
|
-
terrain analyze "What is the state of our test system?"
|
|
56
|
-
terrain insights "What should we fix in our test system?"
|
|
57
|
-
terrain impact "What validations matter for this change?"
|
|
58
|
-
terrain explain "Why did Terrain make this decision?"
|
|
59
|
-
```
|
|
60
|
-
|
|
61
|
-
> **About the example outputs below.** The CLI dumps in this section illustrate the *shape* of Terrain's reports on a large pandas-style repository — they are not literal output from a single live run. A few specific signals shown (`xfailAccumulation` age, statistical flaky-test failure rates, the `0.91+` duplicate similarity threshold) are marked `[experimental]` or `[planned]` in 0.1.2; see [docs/release/feature-status.md](docs/release/feature-status.md) for what's stable, what's experimental, and what's planned. The headline "30 seconds" promise refers to small-to-medium repos (≤ 1,000 test files) on commodity hardware; expect 5–15 seconds on a typical service repo and longer on monorepos.
|
|
62
|
-
|
|
63
|
-
### 1. Analyze — understand the test system
|
|
64
|
-
|
|
65
|
-
```bash
|
|
66
|
-
terrain analyze
|
|
67
|
-
```
|
|
68
|
-
|
|
69
|
-
```
|
|
70
|
-
Terrain — Test Suite Analysis
|
|
71
|
-
============================================================
|
|
72
|
-
|
|
73
|
-
conftest.py fixture fans out to 3,100 tests — any change retriggers the frame/ suite.
|
|
74
|
-
|
|
75
|
-
Key Findings
|
|
76
|
-
------------------------------------------------------------
|
|
77
|
-
1. [HIGH] 23 source files (18%) have low structural coverage
|
|
78
|
-
2. [HIGH] 8 duplicate test clusters with 0.91+ similarity
|
|
79
|
-
3. [MEDIUM] 34 xfail markers older than 180 days
|
|
80
|
-
|
|
81
|
-
Repository Profile
|
|
82
|
-
------------------------------------------------------------
|
|
83
|
-
Test volume: very large
|
|
84
|
-
CI pressure: high
|
|
85
|
-
Coverage confidence: medium
|
|
86
|
-
Redundancy level: medium
|
|
87
|
-
Fanout burden: high
|
|
88
|
-
|
|
89
|
-
Validation Inventory
|
|
90
|
-
------------------------------------------------------------
|
|
91
|
-
Test files: 1,047
|
|
92
|
-
Test cases: 52,341
|
|
93
|
-
Frameworks:
|
|
94
|
-
pytest 1,047 files [unit]
|
|
95
|
-
|
|
96
|
-
Risk Posture
|
|
97
|
-
------------------------------------------------------------
|
|
98
|
-
health: MODERATE
|
|
99
|
-
coverage_depth: ELEVATED
|
|
100
|
-
coverage_diversity: STRONG
|
|
101
|
-
structural_risk: STRONG
|
|
102
|
-
operational_risk: STRONG
|
|
103
|
-
|
|
104
|
-
Next steps:
|
|
105
|
-
terrain insights prioritized actions and recommendations
|
|
106
|
-
terrain impact what tests matter for this change?
|
|
107
|
-
```
|
|
108
|
-
|
|
109
|
-
### 2. Insights — find what to improve
|
|
110
|
-
|
|
111
|
-
```bash
|
|
112
|
-
terrain insights
|
|
113
|
-
```
|
|
114
|
-
|
|
115
|
-
```
|
|
116
|
-
Terrain — Test System Health Report
|
|
117
|
-
============================================================
|
|
118
|
-
|
|
119
|
-
Health Grade: C
|
|
120
|
-
|
|
121
|
-
Reliability Problems (2)
|
|
122
|
-
[HIGH] 12 flaky tests with >10% failure rate
|
|
123
|
-
[MEDIUM] 34 skipped tests consuming CI resources
|
|
124
|
-
|
|
125
|
-
Coverage Debt (2)
|
|
126
|
-
[HIGH] 23 source files (18%) have low structural coverage
|
|
127
|
-
[MEDIUM] conftest.py fixture fans out to 3,100 tests
|
|
128
|
-
|
|
129
|
-
Recommended Actions
|
|
130
|
-
1. [reliability] Quarantine 12 flaky tests
|
|
131
|
-
why: Flaky tests block unrelated PRs and erode CI trust.
|
|
132
|
-
2. [coverage] Add tests for 23 uncovered source files
|
|
133
|
-
why: Changes in uncovered files cannot trigger test selection.
|
|
134
|
-
3. [optimization] Consolidate 8 duplicate test clusters
|
|
135
|
-
why: 0.91+ similarity — redundant CI cost with no coverage benefit.
|
|
136
|
-
```
|
|
137
|
-
|
|
138
|
-
### 3. Impact — understand what a change affects
|
|
139
|
-
|
|
140
|
-
```bash
|
|
141
|
-
terrain impact --base main
|
|
142
|
-
```
|
|
143
|
-
|
|
144
|
-
```
|
|
145
|
-
Terrain Impact Analysis
|
|
146
|
-
============================================================
|
|
147
|
-
|
|
148
|
-
Summary: 3 file(s) changed, 41 test(s) relevant. Posture: needs_attention.
|
|
149
|
-
|
|
150
|
-
Impacted tests: 127 of 52,341 total
|
|
151
|
-
Coverage confidence: High
|
|
152
|
-
|
|
153
|
-
Recommended Tests (41)
|
|
154
|
-
------------------------------------------------------------
|
|
155
|
-
tests/groupby/test_groupby.py [exact]
|
|
156
|
-
Covers: groupby.py:GroupBy.aggregate
|
|
157
|
-
tests/groupby/test_apply.py [exact]
|
|
158
|
-
Covers: groupby.py:GroupBy.apply
|
|
159
|
-
tests/resample/test_base.py [inferred]
|
|
160
|
-
Reached via shared fixture path
|
|
161
|
-
...and 38 more
|
|
162
|
-
|
|
163
|
-
Protection Gaps
|
|
164
|
-
------------------------------------------------------------
|
|
165
|
-
[medium] pandas/core/groupby/ops.py — no covering tests found
|
|
166
|
-
|
|
167
|
-
Next steps:
|
|
168
|
-
terrain impact --show tests full test list
|
|
169
|
-
terrain impact --show gaps all protection gaps
|
|
170
|
-
```
|
|
171
|
-
|
|
172
|
-
### 4. Explain — understand why
|
|
173
|
-
|
|
174
|
-
```bash
|
|
175
|
-
terrain explain tests/io/json/test_pandas.py
|
|
176
|
-
```
|
|
177
|
-
|
|
178
|
-
```
|
|
179
|
-
Test File: tests/io/json/test_pandas.py
|
|
180
|
-
Framework: pytest
|
|
181
|
-
Tests: 84 Assertions: 312
|
|
182
|
-
|
|
183
|
-
Signals (3):
|
|
184
|
-
[high] networkDependency: 12 tests use @pytest.mark.network — flaky in CI
|
|
185
|
-
[medium] weakAssertion: 8 bare assert statements without descriptive messages
|
|
186
|
-
[low] xfailAccumulation: 3 xfail markers older than 180 days
|
|
187
|
-
|
|
188
|
-
Next steps:
|
|
189
|
-
terrain explain selection explain overall test selection strategy
|
|
190
|
-
terrain impact --show tests see all impacted tests
|
|
191
|
-
```
|
|
192
|
-
|
|
193
|
-
See [Canonical User Journeys](docs/product/canonical-user-journeys.md) for the full workflow specification and [example outputs](docs/examples/) for detailed report samples.
|
|
194
|
-
|
|
195
|
-
## Product Philosophy
|
|
196
|
-
|
|
197
|
-
**Inference first.** Terrain reads code. It parses imports, detects frameworks, resolves coverage relationships, and builds dependency graphs from what already exists in the repository. No annotations, no test tagging, no SDK integration required.
|
|
198
|
-
|
|
199
|
-
**Zero-config by default.** `terrain analyze` works on any repository with test files. Coverage data, runtime artifacts, ownership files, and policy rules are optional inputs that enrich the model — but the core analysis requires nothing beyond the code itself.
|
|
200
|
-
|
|
201
|
-
**Explainability over magic.** Every finding carries evidence: which signal type, what confidence level, what dependency path, what scoring rule. `terrain explain` exposes the full reasoning chain behind any decision. Teams should never wonder *why* Terrain said something.
|
|
202
|
-
|
|
203
|
-
**Conservative under uncertainty.** When Terrain encounters ambiguity — a dependency path with low confidence, a file that might or might not be a test — it flags the uncertainty rather than guessing. Impact analysis uses fallback policies with explicit confidence penalties rather than silently expanding scope.
|
|
204
|
-
|
|
205
|
-
**System health, not individual productivity.** Terrain measures the test system. It never attributes quality to individual developers. Ownership information is used for routing and triage, not scoring.
|
|
206
|
-
|
|
207
|
-
## Who Uses Terrain
|
|
208
|
-
|
|
209
|
-
Terrain is framework-agnostic and language-aware. The same analysis model applies across:
|
|
210
|
-
|
|
211
|
-
- **Frontend teams** — React/Vue component tests, Playwright/Cypress E2E suites, Vitest/Jest unit tests
|
|
212
|
-
- **Backend teams** — Go test suites, pytest collections, JUnit hierarchies, integration test infrastructure
|
|
213
|
-
- **Mobile teams** — cross-platform test suites with standard test frameworks
|
|
214
|
-
- **QA / SDET** — test portfolio management, coverage gap analysis, migration planning across frameworks
|
|
215
|
-
- **SRE / Platform** — CI optimization, test selection for pipelines, policy enforcement
|
|
216
|
-
- **AI / ML evaluation** — evaluation suite structure, benchmark test management, coverage across model behaviors
|
|
217
|
-
|
|
218
|
-
The structural model is the same. The signals and recommendations adapt to the framework and test patterns detected.
|
|
219
|
-
|
|
220
|
-
## AI Testing in CI
|
|
221
|
-
|
|
222
|
-
Terrain gives AI components the same CI safety net as regular tests:
|
|
223
|
-
|
|
224
|
-
- **Surface discovery** — automatically detects prompts, contexts, datasets, tool definitions, RAG pipelines, and eval scenarios in your code
|
|
225
|
-
- **Impact-scoped selection** — `terrain ai run --base main` runs only the eval scenarios affected by your change
|
|
226
|
-
- **Protection gaps** — `terrain pr` flags changed AI surfaces that have no eval scenario covering them
|
|
227
|
-
- **Policy enforcement** — block PRs that modify uncovered AI surfaces, regress accuracy, or trigger safety failures
|
|
228
|
-
- **GitHub Action** — drop-in `terrain-ai.yml` workflow template for AI CI gates
|
|
229
|
-
|
|
230
|
-
The same structural graph that powers test selection for regular code also traces AI surface dependencies, so a change to a prompt template triggers the right eval scenarios automatically.
|
|
231
|
-
|
|
232
|
-
## How CI Optimization Emerges
|
|
233
|
-
|
|
234
|
-
Terrain does not start with CI optimization. It starts with understanding.
|
|
235
|
-
|
|
236
|
-
When you run `terrain analyze`, Terrain builds a structural model: which tests exist, which source files they cover, how they depend on shared fixtures, where duplication lives. From that model, CI optimization *emerges*:
|
|
237
|
-
|
|
238
|
-
- **Test selection** — `terrain impact` traces a diff through the dependency graph and returns only the tests that structurally matter, ranked by confidence. This is not a heuristic skip list — it is a graph traversal with evidence.
|
|
239
|
-
- **Redundancy reduction** — `terrain insights` surfaces duplicate test clusters. Removing or consolidating them directly reduces CI time without reducing coverage.
|
|
240
|
-
- **Fanout control** — High-fanout fixtures that trigger thousands of tests on any change are identified and prioritized for splitting.
|
|
241
|
-
- **Confidence-based runs** — Impact analysis assigns confidence scores. CI pipelines can run high-confidence tests immediately and defer low-confidence tests to nightly runs.
|
|
242
|
-
- **What to test next** — `terrain insights` ranks untested source files by dependency count, telling you which test to write first for maximum impact.
|
|
243
|
-
|
|
244
|
-
The result is faster CI that comes from *understanding the test system*, not from skipping tests and hoping for the best.
|
|
245
|
-
|
|
246
|
-
## Installation
|
|
247
|
-
|
|
248
|
-
### Homebrew (macOS and Linux)
|
|
33
|
+
## Install
|
|
249
34
|
|
|
250
35
|
```bash
|
|
36
|
+
# macOS / Linux
|
|
251
37
|
brew install pmclSF/terrain/mapterrain
|
|
252
|
-
```
|
|
253
|
-
|
|
254
|
-
After the first install, you can also tap once and use the short formula name:
|
|
255
|
-
|
|
256
|
-
```bash
|
|
257
|
-
brew tap pmclSF/terrain
|
|
258
|
-
brew install mapterrain
|
|
259
|
-
```
|
|
260
38
|
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
```bash
|
|
39
|
+
# npm (Node 22+ required; macOS/Linux amd64+arm64, Windows amd64)
|
|
264
40
|
npm install -g mapterrain
|
|
265
|
-
```
|
|
266
|
-
|
|
267
|
-
### Go install
|
|
268
41
|
|
|
269
|
-
|
|
42
|
+
# Go
|
|
270
43
|
go install github.com/pmclSF/terrain/cmd/terrain@latest
|
|
271
44
|
```
|
|
272
45
|
|
|
273
|
-
|
|
46
|
+
Pre-built archives are available for macOS and Linux on amd64/arm64, plus Windows on amd64, from the [releases page](https://github.com/pmclSF/terrain/releases). Each release is signed with Sigstore + cosign; see [SECURITY-DATA-HANDLING.md](SECURITY-DATA-HANDLING.md) for verification details.
|
|
274
47
|
|
|
275
|
-
|
|
48
|
+
Package names vary by distribution, but the installed CLI is always `terrain`. The npm package is `mapterrain`; the Homebrew formula is `pmclSF/terrain/mapterrain`.
|
|
276
49
|
|
|
277
|
-
|
|
50
|
+
## Get started
|
|
278
51
|
|
|
279
52
|
```bash
|
|
280
|
-
|
|
281
|
-
|
|
53
|
+
cd your-repo
|
|
54
|
+
terrain analyze # What's the state of our AI + test system?
|
|
55
|
+
terrain report pr # What does this change put at risk?
|
|
282
56
|
```
|
|
283
57
|
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
### Build from source
|
|
58
|
+
Source analysis covers Python, TypeScript/JavaScript, Go, and Java in 0.3.0. Ruby source is not analyzed in 0.3.0, but Ruby/RSpec and other ecosystems can still contribute dependency, runtime, coverage, and eval artifacts, so mixed-language repos get useful signal. No config required; optional artifacts sharpen findings when present.
|
|
287
59
|
|
|
288
|
-
|
|
289
|
-
git clone https://github.com/pmclSF/terrain.git
|
|
290
|
-
cd terrain
|
|
291
|
-
go build -o terrain ./cmd/terrain
|
|
292
|
-
```
|
|
60
|
+
## What it catches
|
|
293
61
|
|
|
294
|
-
|
|
62
|
+
Terrain models the AI surface alongside the test surface and looks for drift across them:
|
|
295
63
|
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
64
|
+
- **Prompt-schema drift** — prompts that reference fields renamed in a schema living in a different language
|
|
65
|
+
- **Hardcoded API keys** — provider-shaped secrets in source (OpenAI, Anthropic, AWS, GCP, etc.)
|
|
66
|
+
- **Eval coverage gaps** — AI surfaces (prompts, agents, RAG pipelines) with no scenario covering them
|
|
67
|
+
- **Model deprecations** — deprecated model IDs lingering in production paths
|
|
68
|
+
- **Cross-language edges** — TS/JS ↔ Python/Go/Java via OpenAPI, tRPC, gRPC, GraphQL, HTTP routes
|
|
69
|
+
- **Framework-migration blockers** — Jest ↔ Vitest, JUnit 4 ↔ 5, Cypress ↔ Playwright
|
|
70
|
+
- **Portfolio drift across repos** — manifest-backed `terrain portfolio --from` rollups show framework-of-record drift across a polyrepo
|
|
71
|
+
- **Untested exports + weak assertions** — public API surfaces with no covering test; assertions that pass on too much
|
|
72
|
+
- **Fixture fanout + duplicate clusters + skip debt** — structural problems that erode CI signal
|
|
299
73
|
|
|
300
|
-
|
|
74
|
+
Each finding carries a stable rule ID, severity, confidence, evidence, and documented remediation. Run `terrain explain <rule-id>` for the long form.
|
|
301
75
|
|
|
302
|
-
|
|
303
|
-
# Detect coverage/runtime data paths (recommended first step)
|
|
304
|
-
terrain init
|
|
76
|
+
## What it looks like
|
|
305
77
|
|
|
306
|
-
|
|
307
|
-
|
|
78
|
+
```
|
|
79
|
+
Terrain · Test Suite Analysis
|
|
80
|
+
────────────────────────────────────────────────────────────
|
|
308
81
|
|
|
309
|
-
|
|
310
|
-
terrain analyze --json
|
|
82
|
+
conftest.py fixture fans out to 3,100 tests — any change retriggers the frame/ suite.
|
|
311
83
|
|
|
312
|
-
|
|
313
|
-
|
|
84
|
+
Key Findings
|
|
85
|
+
────────────────────────────────────────────────────────────
|
|
86
|
+
1. [HIGH] 23 exported code units have no linked tests
|
|
87
|
+
2. [MED] 12 test files have weak assertion density
|
|
88
|
+
3. [LOW] 7 skipped-test patterns need review
|
|
314
89
|
|
|
315
|
-
|
|
316
|
-
|
|
90
|
+
Risk Posture
|
|
91
|
+
────────────────────────────────────────────────────────────
|
|
92
|
+
health: Moderate
|
|
93
|
+
coverage_depth: Elevated
|
|
94
|
+
coverage_diversity: Strong
|
|
95
|
+
structural_risk: Strong
|
|
96
|
+
Signals: 65 (8 high, 34 medium, 23 low)
|
|
317
97
|
```
|
|
318
98
|
|
|
319
|
-
|
|
99
|
+
> Representative output from a large pandas-style repository. Format and labels are stable across runs; the specific numbers vary by repo. Full sample reports for `analyze`, `insights`, `impact`, and `explain` are in [docs/examples/](docs/examples/).
|
|
320
100
|
|
|
321
|
-
|
|
101
|
+
## Workflow
|
|
322
102
|
|
|
323
103
|
| Command | Question |
|
|
324
104
|
|---------|----------|
|
|
325
105
|
| `terrain analyze` | What is the state of our test system? |
|
|
326
|
-
| `terrain
|
|
327
|
-
| `terrain
|
|
328
|
-
| `terrain
|
|
329
|
-
|
|
330
|
-
### Supporting commands
|
|
331
|
-
|
|
332
|
-
| Command | Purpose |
|
|
333
|
-
|---------|---------|
|
|
334
|
-
| `terrain init` | Detect data files and print recommended analyze command |
|
|
335
|
-
| `terrain summary` | Executive summary with risk, trends, benchmark readiness |
|
|
336
|
-
| `terrain focus` | Prioritized next actions |
|
|
337
|
-
| `terrain posture` | Detailed posture breakdown with measurement evidence |
|
|
338
|
-
| `terrain portfolio` | Portfolio intelligence: cost, breadth, leverage, redundancy |
|
|
339
|
-
| `terrain metrics` | Aggregate metrics scorecard |
|
|
340
|
-
| `terrain compare` | Compare two snapshots for trend tracking |
|
|
341
|
-
| `terrain select-tests` | Recommend protective test set for a change |
|
|
342
|
-
| `terrain pr` | PR/change-scoped analysis |
|
|
343
|
-
| `terrain show <type> <id>` | Drill into test, unit, owner, or finding |
|
|
344
|
-
| `terrain migration <sub>` | Migration readiness, blockers, or preview |
|
|
345
|
-
| `terrain policy check` | Evaluate local policy rules |
|
|
346
|
-
| `terrain export benchmark` | Privacy-safe JSON export for benchmarking |
|
|
347
|
-
| `terrain serve` | Local HTTP server with HTML report and JSON API |
|
|
348
|
-
|
|
349
|
-
### AI / eval
|
|
350
|
-
|
|
351
|
-
| Command | Purpose |
|
|
352
|
-
|---------|---------|
|
|
353
|
-
| `terrain ai list` | List detected scenarios, prompts, datasets, eval files |
|
|
354
|
-
| `terrain ai doctor` | Validate AI/eval setup and surface configuration issues |
|
|
355
|
-
| `terrain ai run` | Execute eval scenarios and collect results |
|
|
356
|
-
| `terrain ai replay` | Replay and verify a previous eval run artifact |
|
|
357
|
-
| `terrain ai record` | Record eval results as a baseline snapshot |
|
|
358
|
-
| `terrain ai baseline` | Manage eval baselines (show, compare) |
|
|
359
|
-
|
|
360
|
-
### Conversion / migration
|
|
361
|
-
|
|
362
|
-
| Command | Purpose |
|
|
363
|
-
|---------|---------|
|
|
364
|
-
| `terrain convert <source>` | Go-native test conversion (25 directions) |
|
|
365
|
-
| `terrain convert-config <source>` | Convert framework config files |
|
|
366
|
-
| `terrain migrate <dir>` | Project-wide migration with state tracking |
|
|
367
|
-
| `terrain estimate <dir>` | Estimate migration complexity |
|
|
368
|
-
| `terrain status` | Show migration progress |
|
|
369
|
-
| `terrain checklist` | Generate migration checklist |
|
|
370
|
-
| `terrain doctor [path]` | Run migration diagnostics |
|
|
371
|
-
| `terrain reset` | Clear migration state |
|
|
372
|
-
| `terrain list-conversions` | List supported conversion directions |
|
|
373
|
-
| `terrain shorthands` | List shorthand aliases (e.g., `cy2pw`, `jest2vt`) |
|
|
374
|
-
| `terrain detect <file-or-dir>` | Detect dominant framework |
|
|
375
|
-
|
|
376
|
-
### Advanced / debug
|
|
377
|
-
|
|
378
|
-
| Command | Purpose |
|
|
379
|
-
|---------|---------|
|
|
380
|
-
| `terrain debug graph` | Dependency graph statistics |
|
|
381
|
-
| `terrain debug coverage` | Structural coverage analysis |
|
|
382
|
-
| `terrain debug fanout` | High-fanout node analysis |
|
|
383
|
-
| `terrain debug duplicates` | Duplicate test cluster analysis |
|
|
384
|
-
| `terrain debug depgraph` | Full dependency graph analysis (all engines) |
|
|
385
|
-
|
|
386
|
-
Repository-scoped commands support `--root PATH`, and machine-readable commands support `--json`. Most analysis commands support `--verbose` for additional detail. Run `terrain <command> --help` for full flag documentation.
|
|
387
|
-
|
|
388
|
-
## Architecture Overview
|
|
389
|
-
|
|
390
|
-
Terrain is built around a signal-first architecture:
|
|
106
|
+
| `terrain report pr` | What does this change put at risk? |
|
|
107
|
+
| `terrain report insights` | What should we fix? |
|
|
108
|
+
| `terrain report impact` | What validations matter for this change? |
|
|
109
|
+
| `terrain report explain <target>` | Why did Terrain make this decision? |
|
|
391
110
|
|
|
392
|
-
|
|
393
|
-
Repository scan → Signal detection → Risk modeling → Reporting
|
|
394
|
-
│ │ │ │
|
|
395
|
-
test files framework-specific explainable human-readable
|
|
396
|
-
source files pattern detectors risk scoring + JSON output
|
|
397
|
-
coverage data quality signals with evidence
|
|
398
|
-
runtime artifacts health signals chains
|
|
399
|
-
ownership files migration signals
|
|
400
|
-
policy rules governance signals
|
|
401
|
-
```
|
|
111
|
+
The bare forms (`terrain insights`, `terrain impact`, `terrain explain`) work as aliases. AI / eval verbs, framework conversion, debug drill-downs, and the slash-receiver round out the surface — full reference in [docs/cli-spec.md](docs/cli-spec.md).
|
|
402
112
|
|
|
403
|
-
|
|
404
|
-
- **Snapshots** (`TestSuiteSnapshot`) are the canonical serialized artifact — the complete structural model of a test system at a point in time
|
|
405
|
-
- **Risk surfaces** are derived from signals with explainable scoring across dimensions (quality, reliability, speed, governance)
|
|
406
|
-
- **Dependency graphs** model import relationships, fixture fanout, and structural coverage
|
|
407
|
-
- **Reports** synthesize signals, risk, trends, and benchmark readiness into actionable output
|
|
113
|
+
## CI integration
|
|
408
114
|
|
|
409
|
-
```
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
115
|
+
```yaml
|
|
116
|
+
# .github/workflows/terrain.yml
|
|
117
|
+
name: terrain
|
|
118
|
+
on: pull_request
|
|
119
|
+
|
|
120
|
+
jobs:
|
|
121
|
+
analyze:
|
|
122
|
+
runs-on: ubuntu-latest
|
|
123
|
+
steps:
|
|
124
|
+
- uses: actions/checkout@v6
|
|
125
|
+
with: { fetch-depth: 0 }
|
|
126
|
+
- uses: actions/setup-node@v6
|
|
127
|
+
with: { node-version: '22.x' }
|
|
128
|
+
- run: npm install -g mapterrain
|
|
129
|
+
- run: |
|
|
130
|
+
terrain test \
|
|
131
|
+
--junit terrain-results.xml \
|
|
132
|
+
--summary "$GITHUB_STEP_SUMMARY"
|
|
413
133
|
```
|
|
414
134
|
|
|
415
|
-
|
|
135
|
+
`terrain test` is the CI-mode wrapper. The JUnit XML lets your CI render Terrain findings as test cases; setting `--summary "$GITHUB_STEP_SUMMARY"` makes them appear on the workflow run page automatically.
|
|
416
136
|
|
|
417
|
-
|
|
137
|
+
For a blocking gate, add `--fail-on=high` (or `--fail-on=critical` for the strictest threshold). For onboarding a repo with existing debt, pair with `--new-findings-only --baseline <path>` so only regressions block the build. To restrict the gate to AI-related changes, add a `paths:` filter on prompt / schema / Python / TS file globs (see [docs/examples/gate/](docs/examples/gate/) for the full templates).
|
|
418
138
|
|
|
419
|
-
|
|
139
|
+
## Boundaries
|
|
420
140
|
|
|
421
|
-
|
|
422
|
-
# Save a snapshot
|
|
423
|
-
terrain analyze --write-snapshot
|
|
141
|
+
Terrain is intentionally a CI/local pre-flight layer. It composes with the tools teams already trust instead of replacing them:
|
|
424
142
|
|
|
425
|
-
|
|
426
|
-
|
|
143
|
+
- It reads what `pytest`, `jest`, `go test`, Playwright, Promptfoo, DeepEval, Ragas, Great Expectations, and Gauntlet-style eval-result artifacts produce. The AI namespace can execute supported eval-framework commands for `terrain ai run`, but Terrain does not replace those frameworks.
|
|
144
|
+
- It ingests coverage reports when you have them; it does not instrument code.
|
|
145
|
+
- It complements Semgrep / CodeQL / Sonar rather than replacing application-code bug finding.
|
|
146
|
+
- It routes ownership data to findings; it never produces leaderboards or per-developer scores.
|
|
147
|
+
- It runs locally with zero outbound network calls during analysis — verifiable with `terrain --print-network`. The install paths download signed binaries from GitHub Releases; Terrain itself does not upload analysis results.
|
|
148
|
+
- CI artifacts and MCP responses can include repo paths, line numbers, diagnostic text, and sometimes source excerpts. Treat the CI platform or AI assistant you connect as part of the trust boundary.
|
|
427
149
|
|
|
428
|
-
|
|
429
|
-
terrain compare
|
|
150
|
+
Under the hood, Terrain combines source analysis, dependency-graph construction, artifact ingestion, and deterministic rule evaluation. The product is the CI gate and protection graph; static analysis is one mechanism inside it.
|
|
430
151
|
|
|
431
|
-
|
|
432
|
-
terrain summary
|
|
433
|
-
```
|
|
152
|
+
## AI-aware testing
|
|
434
153
|
|
|
435
|
-
|
|
154
|
+
The same dependency graph that powers test selection for application code also traces AI surface edges. When Terrain can map a prompt-template, schema, or eval-artifact change to declared or inferred scenarios, it selects the impacted evals automatically.
|
|
436
155
|
|
|
437
|
-
|
|
156
|
+
- `terrain ai run --base main` — run only the evals your change affects
|
|
157
|
+
- `terrain report pr` — flag changed AI surfaces with no covering eval
|
|
158
|
+
- `terrain ai findings` — AI eval-gap findings with evidence chains
|
|
438
159
|
|
|
439
|
-
|
|
160
|
+
Two generators help adopters harden a prompt before deploying it:
|
|
440
161
|
|
|
441
|
-
```
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
max_weak_assertions: 10
|
|
445
|
-
max_mock_heavy_tests: 5
|
|
162
|
+
```bash
|
|
163
|
+
terrain inject --prompt prompts/main.md # generate jailbreak-shaped test inputs
|
|
164
|
+
terrain scaffold --schema schemas/input.json # generate boundary-case mutation tests
|
|
446
165
|
```
|
|
447
166
|
|
|
448
|
-
|
|
167
|
+
Both emit runnable pytest / vitest scaffolds you drop into your test tree. Terrain never calls the model; the assertion is yours.
|
|
449
168
|
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
terrain
|
|
453
|
-
```
|
|
169
|
+
## MCP integration
|
|
170
|
+
|
|
171
|
+
`terrain mcp` exposes the last analyze run to AI coding assistants (Claude Code, Cursor, others) over the [Model Context Protocol](https://modelcontextprotocol.io). The assistant can query findings, drill into surfaces, and read baselines without you copy-pasting JSON — so "why is this PR failing Terrain?" gets a useful answer in the IDE instead of a context-switch to the terminal.
|
|
454
172
|
|
|
455
|
-
|
|
173
|
+
The MCP server is local and read-only. The assistant client decides what context, if any, is sent to its model provider.
|
|
174
|
+
|
|
175
|
+
## Plugins
|
|
176
|
+
|
|
177
|
+
Third-party detectors ship as YAML manifests; `terrain plugins manifest <path>` validates one against the stable schema. The plugin runtime — the loader that executes registered detectors — is reserved for a future release. Adopters can author and publish manifests now and they'll be loadable when the runtime ships. See [`examples/plugins/example-manifest.yaml`](examples/plugins/example-manifest.yaml).
|
|
456
178
|
|
|
457
179
|
## Documentation
|
|
458
180
|
|
|
459
|
-
|
|
460
|
-
- [CLI Specification](docs/cli-spec.md) — full command and flag reference
|
|
461
|
-
- [Example Reports](docs/examples/) — analyze, impact, insights, explain output samples
|
|
462
|
-
- [Canonical User Journeys](docs/product/canonical-user-journeys.md) — primary workflows and expected outcomes
|
|
463
|
-
- [Signal Model](docs/signal-model.md) — the core signal abstraction
|
|
464
|
-
- [Architecture](docs/architecture/) — design documents and technical specifications
|
|
465
|
-
- [Contributing](CONTRIBUTING.md) — how to build, test, and extend Terrain
|
|
181
|
+
**Get started**
|
|
466
182
|
|
|
467
|
-
|
|
183
|
+
- [Quickstart](docs/quickstart.md) — first report in 5 minutes
|
|
184
|
+
- [CLI specification](docs/cli-spec.md) — full command + flag reference
|
|
185
|
+
- [Example reports](docs/examples/) — analyze / impact / insights / explain samples
|
|
468
186
|
|
|
469
|
-
|
|
470
|
-
# Build
|
|
471
|
-
go build -o terrain ./cmd/terrain
|
|
187
|
+
**Reference**
|
|
472
188
|
|
|
473
|
-
|
|
474
|
-
|
|
189
|
+
- [Design](DESIGN.md) — architecture, package map, signal pipeline
|
|
190
|
+
- [Severity rubric](docs/severity-rubric.md) — severity labels and configuration
|
|
191
|
+
- [Compatibility](docs/compatibility.md) — supported OSes, Go versions, frameworks, schemas
|
|
192
|
+
- [Glossary](docs/glossary.md) — Terrain-specific vocabulary
|
|
193
|
+
- [Versioning](docs/versioning.md) — what counts as a breaking change
|
|
475
194
|
|
|
476
|
-
|
|
477
|
-
|
|
478
|
-
|
|
195
|
+
**Project**
|
|
196
|
+
|
|
197
|
+
- [CHANGELOG](CHANGELOG.md) — release history
|
|
198
|
+
- [Security](SECURITY.md) — supported versions + vulnerability disclosure
|
|
199
|
+
- [Contributing](CONTRIBUTING.md) — how to build, test, and extend Terrain
|
|
479
200
|
|
|
480
201
|
## License
|
|
481
202
|
|
|
482
|
-
Apache License 2.0 — see [LICENSE](LICENSE)
|
|
203
|
+
Apache License 2.0 — see [LICENSE](LICENSE).
|
package/SECURITY.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
## Supported Versions
|
|
4
4
|
|
|
5
|
-
Security updates are provided for the latest published major version of Terrain
|
|
5
|
+
Security updates are provided for the latest published major version of Terrain (the `terrain` Go CLI, distributed via Homebrew, npm, and direct binary).
|
|
6
6
|
|
|
7
7
|
## Reporting a Vulnerability
|
|
8
8
|
|
package/bin/postinstall.js
CHANGED
|
@@ -1,12 +1,42 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
|
|
3
|
-
import {
|
|
3
|
+
import {
|
|
4
|
+
ensureTerrainBinary,
|
|
5
|
+
writeInstallFailureMarker,
|
|
6
|
+
clearInstallFailureMarker,
|
|
7
|
+
} from './terrain-installer.js';
|
|
4
8
|
|
|
9
|
+
// We intentionally don't fail npm install when the binary fetch fails —
|
|
10
|
+
// CI pipelines that run `npm install` as part of a larger flow can
|
|
11
|
+
// recover from a transient download issue, and forcing every cosign-
|
|
12
|
+
// missing host to fail the install would be more disruptive than the
|
|
13
|
+
// failure mode itself. But a silent warning is also wrong: a missing
|
|
14
|
+
// binary should not be discovered five minutes later when the user
|
|
15
|
+
// runs `terrain analyze` and gets a confusing retry.
|
|
16
|
+
//
|
|
17
|
+
// Compromise: write a marker file describing the failure. The CLI
|
|
18
|
+
// trampoline reads it on first run and prints a clear, framed error
|
|
19
|
+
// pointing at the remediation (install cosign, or set the documented
|
|
20
|
+
// opt-out env var) instead of attempting a silent retry.
|
|
5
21
|
try {
|
|
6
22
|
await ensureTerrainBinary({ quiet: false });
|
|
23
|
+
// Clean up any stale marker from a previous failed install.
|
|
24
|
+
await clearInstallFailureMarker();
|
|
7
25
|
} catch (error) {
|
|
26
|
+
await writeInstallFailureMarker(error);
|
|
8
27
|
process.stderr.write(
|
|
9
|
-
|
|
10
|
-
'
|
|
28
|
+
'\n' +
|
|
29
|
+
'!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n' +
|
|
30
|
+
'! mapterrain: binary install FAILED !\n' +
|
|
31
|
+
'!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n' +
|
|
32
|
+
'\n' +
|
|
33
|
+
`${error.message}\n` +
|
|
34
|
+
'\n' +
|
|
35
|
+
'npm install reports success, but the `terrain` binary is NOT\n' +
|
|
36
|
+
'installed. Running `terrain` will fail with the same error\n' +
|
|
37
|
+
'until the underlying issue is resolved.\n' +
|
|
38
|
+
'\n' +
|
|
39
|
+
'Marker written to ~/.terrain/install-failure.log\n' +
|
|
40
|
+
'\n'
|
|
11
41
|
);
|
|
12
42
|
}
|
package/bin/terrain-installer.js
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
|
|
3
3
|
import { execFileSync, spawn } from 'child_process';
|
|
4
|
+
import { createHash } from 'crypto';
|
|
4
5
|
import { createWriteStream, existsSync } from 'fs';
|
|
5
6
|
import fs from 'fs/promises';
|
|
6
7
|
import https from 'https';
|
|
@@ -17,8 +18,85 @@ const packageJson = JSON.parse(
|
|
|
17
18
|
|
|
18
19
|
const GITHUB_OWNER = 'pmclSF';
|
|
19
20
|
const GITHUB_REPO = 'terrain';
|
|
21
|
+
const DOWNLOAD_TIMEOUT_MS = 120000;
|
|
22
|
+
|
|
23
|
+
// installFailureMarkerPath returns the path where a failed install
|
|
24
|
+
// records its error. The CLI trampoline checks this before retrying
|
|
25
|
+
// so users see a clear remediation message instead of a confusing
|
|
26
|
+
// retry of the same failure. ~/.terrain is also where local snapshots
|
|
27
|
+
// live, so the location is already a Terrain working directory.
|
|
28
|
+
function installFailureMarkerPath() {
|
|
29
|
+
return path.join(os.homedir(), '.terrain', 'install-failure.log');
|
|
30
|
+
}
|
|
20
31
|
|
|
21
|
-
|
|
32
|
+
// writeInstallFailureMarker is called from postinstall.js when
|
|
33
|
+
// `npm install` fails to fetch / verify the binary. It captures the
|
|
34
|
+
// error so the next `terrain` invocation can print it verbatim
|
|
35
|
+
// without attempting another silent retry.
|
|
36
|
+
export async function writeInstallFailureMarker(error) {
|
|
37
|
+
try {
|
|
38
|
+
const markerPath = installFailureMarkerPath();
|
|
39
|
+
await fs.mkdir(path.dirname(markerPath), { recursive: true });
|
|
40
|
+
const body = JSON.stringify(
|
|
41
|
+
{
|
|
42
|
+
timestamp: new Date().toISOString(),
|
|
43
|
+
message: error?.message ?? String(error),
|
|
44
|
+
stack: error?.stack ?? null,
|
|
45
|
+
platform: `${process.platform}/${process.arch}`,
|
|
46
|
+
version: packageJson.version,
|
|
47
|
+
},
|
|
48
|
+
null,
|
|
49
|
+
2
|
|
50
|
+
);
|
|
51
|
+
await fs.writeFile(markerPath, body, 'utf8');
|
|
52
|
+
} catch (writeErr) {
|
|
53
|
+
// Failing to write the marker is itself non-fatal; the postinstall
|
|
54
|
+
// warning has already been printed.
|
|
55
|
+
process.stderr.write(
|
|
56
|
+
`[mapterrain] (could not record install-failure marker: ${writeErr.message})\n`
|
|
57
|
+
);
|
|
58
|
+
}
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
// clearInstallFailureMarker removes the marker on a successful
|
|
62
|
+
// install or successful first run. Idempotent.
|
|
63
|
+
export async function clearInstallFailureMarker() {
|
|
64
|
+
try {
|
|
65
|
+
await fs.unlink(installFailureMarkerPath());
|
|
66
|
+
} catch (err) {
|
|
67
|
+
if (err.code !== 'ENOENT') {
|
|
68
|
+
// ENOENT is the happy path (no marker existed). Anything else
|
|
69
|
+
// is unexpected; surface it but don't fail.
|
|
70
|
+
process.stderr.write(
|
|
71
|
+
`[mapterrain] (could not clear install-failure marker: ${err.message})\n`
|
|
72
|
+
);
|
|
73
|
+
}
|
|
74
|
+
}
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
// readInstallFailureMarker returns the recorded error message, or
|
|
78
|
+
// null if no marker exists.
|
|
79
|
+
async function readInstallFailureMarker() {
|
|
80
|
+
try {
|
|
81
|
+
const body = await fs.readFile(installFailureMarkerPath(), 'utf8');
|
|
82
|
+
return JSON.parse(body);
|
|
83
|
+
} catch (err) {
|
|
84
|
+
return null;
|
|
85
|
+
}
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
const SUPPORTED_TARGETS = new Set([
|
|
89
|
+
'darwin/amd64',
|
|
90
|
+
'darwin/arm64',
|
|
91
|
+
'linux/amd64',
|
|
92
|
+
'linux/arm64',
|
|
93
|
+
'windows/amd64',
|
|
94
|
+
]);
|
|
95
|
+
|
|
96
|
+
export function targetForPlatform(
|
|
97
|
+
platform = process.platform,
|
|
98
|
+
arch = process.arch
|
|
99
|
+
) {
|
|
22
100
|
const goosMap = {
|
|
23
101
|
darwin: 'darwin',
|
|
24
102
|
linux: 'linux',
|
|
@@ -29,12 +107,20 @@ function currentTarget() {
|
|
|
29
107
|
arm64: 'arm64',
|
|
30
108
|
};
|
|
31
109
|
|
|
32
|
-
const goos = goosMap[
|
|
33
|
-
const goarch = goarchMap[
|
|
110
|
+
const goos = goosMap[platform];
|
|
111
|
+
const goarch = goarchMap[arch];
|
|
34
112
|
if (!goos || !goarch) {
|
|
35
113
|
throw new Error(
|
|
36
|
-
`Unsupported platform ${
|
|
37
|
-
'Install Terrain manually from GitHub Releases or
|
|
114
|
+
`Unsupported platform ${platform}/${arch}. ` +
|
|
115
|
+
'Install Terrain manually from GitHub Releases, Homebrew on macOS/Linux, or source.'
|
|
116
|
+
);
|
|
117
|
+
}
|
|
118
|
+
if (!SUPPORTED_TARGETS.has(`${goos}/${goarch}`)) {
|
|
119
|
+
throw new Error(
|
|
120
|
+
`Unsupported prebuilt Terrain target ${goos}/${goarch}. ` +
|
|
121
|
+
'Published npm-install binaries are available for ' +
|
|
122
|
+
`${Array.from(SUPPORTED_TARGETS).sort().join(', ')}. ` +
|
|
123
|
+
'Install Terrain manually from source with `go install github.com/pmclSF/terrain/cmd/terrain@latest`.'
|
|
38
124
|
);
|
|
39
125
|
}
|
|
40
126
|
|
|
@@ -46,6 +132,10 @@ function currentTarget() {
|
|
|
46
132
|
};
|
|
47
133
|
}
|
|
48
134
|
|
|
135
|
+
function currentTarget() {
|
|
136
|
+
return targetForPlatform();
|
|
137
|
+
}
|
|
138
|
+
|
|
49
139
|
function isDevelopmentCheckout(rootDir = packageRoot) {
|
|
50
140
|
return existsSync(path.join(rootDir, 'cmd', 'terrain'));
|
|
51
141
|
}
|
|
@@ -71,11 +161,15 @@ function archiveFileName(version) {
|
|
|
71
161
|
return `terrain_${version}_${target.goos}_${target.goarch}.${target.archiveExt}`;
|
|
72
162
|
}
|
|
73
163
|
|
|
74
|
-
function
|
|
164
|
+
function releaseDownloadBaseUrl(version) {
|
|
75
165
|
const baseUrl =
|
|
76
166
|
process.env.TERRAIN_INSTALLER_BASE_URL ||
|
|
77
167
|
`https://github.com/${GITHUB_OWNER}/${GITHUB_REPO}/releases/download`;
|
|
78
|
-
return `${baseUrl}/v${version}
|
|
168
|
+
return `${baseUrl}/v${version}`;
|
|
169
|
+
}
|
|
170
|
+
|
|
171
|
+
function archiveDownloadUrl(version) {
|
|
172
|
+
return `${releaseDownloadBaseUrl(version)}/${archiveFileName(version)}`;
|
|
79
173
|
}
|
|
80
174
|
|
|
81
175
|
function signatureDownloadUrl(version) {
|
|
@@ -86,6 +180,10 @@ function certificateDownloadUrl(version) {
|
|
|
86
180
|
return `${archiveDownloadUrl(version)}.pem`;
|
|
87
181
|
}
|
|
88
182
|
|
|
183
|
+
function checksumDownloadUrl(version) {
|
|
184
|
+
return `${releaseDownloadBaseUrl(version)}/checksums.txt`;
|
|
185
|
+
}
|
|
186
|
+
|
|
89
187
|
function expectedSignerIdentity(version) {
|
|
90
188
|
// The keyless Sigstore signature is anchored to the GitHub Actions workflow
|
|
91
189
|
// that ran goreleaser at release time. The workflow runs on the v<version>
|
|
@@ -107,13 +205,84 @@ function isCosignAvailable() {
|
|
|
107
205
|
}
|
|
108
206
|
}
|
|
109
207
|
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
208
|
+
export function checksumFromManifest(manifestText, archiveName) {
|
|
209
|
+
for (const rawLine of manifestText.split(/\r?\n/)) {
|
|
210
|
+
const line = rawLine.trim();
|
|
211
|
+
if (!line) {
|
|
212
|
+
continue;
|
|
213
|
+
}
|
|
214
|
+
const match = line.match(/^([a-fA-F0-9]{64})\s+(.+)$/);
|
|
215
|
+
if (!match) {
|
|
216
|
+
continue;
|
|
217
|
+
}
|
|
218
|
+
const candidate = path.basename(match[2].replace(/^\*/, '').trim());
|
|
219
|
+
if (candidate === archiveName) {
|
|
220
|
+
return match[1].toLowerCase();
|
|
221
|
+
}
|
|
222
|
+
}
|
|
223
|
+
return null;
|
|
224
|
+
}
|
|
225
|
+
|
|
226
|
+
export function verifyChecksumDigest(manifestText, archiveName, actualDigest) {
|
|
227
|
+
const expected = checksumFromManifest(manifestText, archiveName);
|
|
228
|
+
if (!expected) {
|
|
229
|
+
throw new Error(
|
|
230
|
+
`checksums.txt did not contain an entry for ${archiveName}`
|
|
231
|
+
);
|
|
232
|
+
}
|
|
233
|
+
const actual = actualDigest.toLowerCase();
|
|
234
|
+
if (actual !== expected) {
|
|
235
|
+
throw new Error(
|
|
236
|
+
`checksum mismatch for ${archiveName}: expected ${expected}, got ${actual}`
|
|
237
|
+
);
|
|
238
|
+
}
|
|
239
|
+
return expected;
|
|
240
|
+
}
|
|
241
|
+
|
|
242
|
+
async function sha256File(filePath) {
|
|
243
|
+
const hash = createHash('sha256');
|
|
244
|
+
hash.update(await fs.readFile(filePath));
|
|
245
|
+
return hash.digest('hex');
|
|
246
|
+
}
|
|
247
|
+
|
|
248
|
+
async function verifyChecksum({ archivePath, version, tempDir, quiet }) {
|
|
249
|
+
const checksumsPath = path.join(tempDir, 'checksums.txt');
|
|
250
|
+
await downloadFile(checksumDownloadUrl(version), checksumsPath);
|
|
251
|
+
|
|
252
|
+
const archiveName = path.basename(archivePath);
|
|
253
|
+
const manifest = await fs.readFile(checksumsPath, 'utf8');
|
|
254
|
+
const actual = await sha256File(archivePath);
|
|
255
|
+
verifyChecksumDigest(manifest, archiveName, actual);
|
|
256
|
+
|
|
257
|
+
log(`Checksum verified for ${archiveName}`, quiet);
|
|
258
|
+
return { verified: true, reason: 'checksum' };
|
|
259
|
+
}
|
|
260
|
+
|
|
261
|
+
// Sigstore signature verification.
|
|
262
|
+
//
|
|
263
|
+
// 0.3.x policy: Sigstore verification is MANDATORY by default. If
|
|
264
|
+
// `cosign` is not available on the host, the install fails with a
|
|
265
|
+
// clear remediation pointer. The escape for trusted/CI/air-gapped
|
|
266
|
+
// environments is the documented opt-out
|
|
267
|
+
// `TERRAIN_INSTALLER_SKIP_VERIFY=1`.
|
|
268
|
+
//
|
|
269
|
+
// Earlier revisions silently degraded to "checksum-only" when cosign
|
|
270
|
+
// was missing, which meant a typical npm-install on a host without
|
|
271
|
+
// cosign (most macOS / Linux dev machines) skipped Sigstore entirely
|
|
272
|
+
// without any signal in the install log beyond a one-line "falling
|
|
273
|
+
// back" message: the strong-integrity guarantee we advertise degraded
|
|
274
|
+
// silently to weak by default. Mandatory verification closes the gap;
|
|
275
|
+
// the env-var escape keeps adoption viable.
|
|
276
|
+
//
|
|
277
|
+
// Escape hatches:
|
|
278
|
+
//
|
|
279
|
+
// - TERRAIN_INSTALLER_SKIP_VERIFY=1 — fully opt out (CI / air-gapped).
|
|
280
|
+
// Prints a WARNING so the bypass is auditable.
|
|
281
|
+
// - TERRAIN_INSTALLER_ALLOW_MISSING_COSIGN=1 — opt-in degrade-to-
|
|
282
|
+
// checksum behavior for hosts that genuinely cannot install
|
|
283
|
+
// cosign.
|
|
114
284
|
//
|
|
115
|
-
//
|
|
116
|
-
// at which point the warning escalates to an error.
|
|
285
|
+
// Once cosign is on the host, every verify failure is a hard error.
|
|
117
286
|
async function verifySignatureBestEffort({
|
|
118
287
|
archivePath,
|
|
119
288
|
version,
|
|
@@ -123,20 +292,38 @@ async function verifySignatureBestEffort({
|
|
|
123
292
|
}) {
|
|
124
293
|
if (env.TERRAIN_INSTALLER_SKIP_VERIFY === '1') {
|
|
125
294
|
log(
|
|
126
|
-
'
|
|
295
|
+
'WARNING: signature verification skipped (TERRAIN_INSTALLER_SKIP_VERIFY=1). ' +
|
|
296
|
+
'Set this only in trusted CI / air-gapped environments where ' +
|
|
297
|
+
'integrity is established by another channel.',
|
|
127
298
|
quiet
|
|
128
299
|
);
|
|
129
300
|
return { verified: false, reason: 'skipped-by-env' };
|
|
130
301
|
}
|
|
131
302
|
|
|
132
303
|
if (!isCosignAvailable()) {
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
'
|
|
136
|
-
|
|
137
|
-
|
|
304
|
+
if (env.TERRAIN_INSTALLER_ALLOW_MISSING_COSIGN === '1') {
|
|
305
|
+
log(
|
|
306
|
+
'cosign not found on PATH. Continuing with checksum-only verification ' +
|
|
307
|
+
'because TERRAIN_INSTALLER_ALLOW_MISSING_COSIGN=1 is set. ' +
|
|
308
|
+
'For stronger integrity guarantees install cosign ' +
|
|
309
|
+
'(https://github.com/sigstore/cosign) and reinstall.',
|
|
310
|
+
quiet
|
|
311
|
+
);
|
|
312
|
+
return await verifyChecksum({ archivePath, version, tempDir, quiet });
|
|
313
|
+
}
|
|
314
|
+
throw new Error(
|
|
315
|
+
'cosign is required to verify the Sigstore signature on the Terrain ' +
|
|
316
|
+
'release archive, but was not found on PATH.\n\n' +
|
|
317
|
+
'Resolve by one of:\n' +
|
|
318
|
+
' 1. Install cosign: https://github.com/sigstore/cosign#installation\n' +
|
|
319
|
+
' (Homebrew: `brew install cosign`. Linux: see release notes.)\n' +
|
|
320
|
+
' 2. If this host genuinely cannot install cosign and you trust the ' +
|
|
321
|
+
'GitHub-provided checksum file, set ' +
|
|
322
|
+
'TERRAIN_INSTALLER_ALLOW_MISSING_COSIGN=1 to fall back to ' +
|
|
323
|
+
'checksum-only verification.\n' +
|
|
324
|
+
' 3. To skip integrity verification entirely (NOT recommended), ' +
|
|
325
|
+
'set TERRAIN_INSTALLER_SKIP_VERIFY=1.'
|
|
138
326
|
);
|
|
139
|
-
return { verified: false, reason: 'cosign-missing' };
|
|
140
327
|
}
|
|
141
328
|
|
|
142
329
|
const sigPath = path.join(tempDir, `${path.basename(archivePath)}.sig`);
|
|
@@ -146,12 +333,14 @@ async function verifySignatureBestEffort({
|
|
|
146
333
|
await downloadFile(signatureDownloadUrl(version), sigPath);
|
|
147
334
|
await downloadFile(certificateDownloadUrl(version), certPath);
|
|
148
335
|
} catch (error) {
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
336
|
+
// Hard error in 0.3: if cosign is present, the signature download
|
|
337
|
+
// is required. The release pipeline produces signatures for every
|
|
338
|
+
// archive; their absence is a real failure mode worth surfacing.
|
|
339
|
+
throw new Error(
|
|
340
|
+
`cosign is installed but the Sigstore signature artifacts for ` +
|
|
341
|
+
`terrain ${version} could not be downloaded: ${error.message}. ` +
|
|
342
|
+
`Set TERRAIN_INSTALLER_SKIP_VERIFY=1 to bypass at your own risk.`
|
|
153
343
|
);
|
|
154
|
-
return { verified: false, reason: 'sig-download-failed' };
|
|
155
344
|
}
|
|
156
345
|
|
|
157
346
|
try {
|
|
@@ -177,14 +366,17 @@ async function verifySignatureBestEffort({
|
|
|
177
366
|
);
|
|
178
367
|
return { verified: true, reason: 'ok' };
|
|
179
368
|
} catch (error) {
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
369
|
+
// Hard error in 0.3: a verify-blob failure means the archive on disk
|
|
370
|
+
// does NOT match the signed certificate. Aborting the install is
|
|
371
|
+
// strictly safer than silently continuing.
|
|
372
|
+
const detail = error.stderr
|
|
373
|
+
? error.stderr.toString().trim()
|
|
374
|
+
: error.message;
|
|
375
|
+
throw new Error(
|
|
376
|
+
`cosign verify-blob FAILED for ${path.basename(archivePath)}: ${detail}. ` +
|
|
377
|
+
`The downloaded archive does not match its Sigstore signature; ` +
|
|
378
|
+
`the binary may have been tampered with. Install aborted.`
|
|
186
379
|
);
|
|
187
|
-
return { verified: false, reason: 'verify-failed' };
|
|
188
380
|
}
|
|
189
381
|
}
|
|
190
382
|
|
|
@@ -206,7 +398,18 @@ function log(message, quiet = false) {
|
|
|
206
398
|
}
|
|
207
399
|
}
|
|
208
400
|
|
|
209
|
-
|
|
401
|
+
// MAX_REDIRECTS caps redirect chains to defend against misconfigured
|
|
402
|
+
// proxies that loop. 5 covers every normal redirect chain (GitHub
|
|
403
|
+
// release → CDN → storage backend) with margin to spare. Without this
|
|
404
|
+
// cap the recursion is unbounded — a redirect loop hangs the installer
|
|
405
|
+
// until the OS kills it.
|
|
406
|
+
const MAX_REDIRECTS = 5;
|
|
407
|
+
|
|
408
|
+
async function downloadFile(
|
|
409
|
+
url,
|
|
410
|
+
destinationPath,
|
|
411
|
+
redirectsRemaining = MAX_REDIRECTS
|
|
412
|
+
) {
|
|
210
413
|
await new Promise((resolve, reject) => {
|
|
211
414
|
const request = https.get(
|
|
212
415
|
url,
|
|
@@ -223,8 +426,22 @@ async function downloadFile(url, destinationPath) {
|
|
|
223
426
|
response.headers.location
|
|
224
427
|
) {
|
|
225
428
|
response.resume();
|
|
429
|
+
if (redirectsRemaining <= 0) {
|
|
430
|
+
reject(
|
|
431
|
+
new Error(
|
|
432
|
+
`download exceeded ${MAX_REDIRECTS} redirects for ${url}; ` +
|
|
433
|
+
'check for proxy redirect loops or set ' +
|
|
434
|
+
'TERRAIN_INSTALLER_BASE_URL to a direct download host.'
|
|
435
|
+
)
|
|
436
|
+
);
|
|
437
|
+
return;
|
|
438
|
+
}
|
|
226
439
|
try {
|
|
227
|
-
await downloadFile(
|
|
440
|
+
await downloadFile(
|
|
441
|
+
response.headers.location,
|
|
442
|
+
destinationPath,
|
|
443
|
+
redirectsRemaining - 1
|
|
444
|
+
);
|
|
228
445
|
resolve();
|
|
229
446
|
} catch (error) {
|
|
230
447
|
reject(error);
|
|
@@ -251,6 +468,13 @@ async function downloadFile(url, destinationPath) {
|
|
|
251
468
|
}
|
|
252
469
|
);
|
|
253
470
|
|
|
471
|
+
request.setTimeout(DOWNLOAD_TIMEOUT_MS, () => {
|
|
472
|
+
request.destroy(
|
|
473
|
+
new Error(
|
|
474
|
+
`download timed out after ${DOWNLOAD_TIMEOUT_MS / 1000}s for ${url}`
|
|
475
|
+
)
|
|
476
|
+
);
|
|
477
|
+
});
|
|
254
478
|
request.on('error', reject);
|
|
255
479
|
});
|
|
256
480
|
}
|
|
@@ -385,6 +609,25 @@ export async function runTerrainCli(argv = process.argv.slice(2)) {
|
|
|
385
609
|
return;
|
|
386
610
|
}
|
|
387
611
|
|
|
612
|
+
// Check for a recorded install failure before attempting a silent
|
|
613
|
+
// retry. If `npm install` failed to fetch/verify the binary, the
|
|
614
|
+
// marker file records the original error; surface it verbatim
|
|
615
|
+
// instead of pretending nothing happened.
|
|
616
|
+
const marker = await readInstallFailureMarker();
|
|
617
|
+
if (marker && !existsSync(installedBinaryPath(rootDir))) {
|
|
618
|
+
throw new Error(
|
|
619
|
+
'Terrain binary is not installed.\n\n' +
|
|
620
|
+
`Recorded install failure (${marker.timestamp}, ${marker.platform}, v${marker.version}):\n` +
|
|
621
|
+
` ${marker.message}\n\n` +
|
|
622
|
+
'Resolve the underlying issue, then either:\n' +
|
|
623
|
+
' - Re-run `npm install -g mapterrain` after installing cosign\n' +
|
|
624
|
+
' - Set TERRAIN_INSTALLER_ALLOW_MISSING_COSIGN=1 to fall back to\n' +
|
|
625
|
+
' checksum-only verification, or\n' +
|
|
626
|
+
' - Set TERRAIN_INSTALLER_SKIP_VERIFY=1 to skip verification entirely.\n\n' +
|
|
627
|
+
'Marker file: ~/.terrain/install-failure.log'
|
|
628
|
+
);
|
|
629
|
+
}
|
|
630
|
+
|
|
388
631
|
let binaryPath;
|
|
389
632
|
try {
|
|
390
633
|
binaryPath = await ensureTerrainBinary({ rootDir });
|
|
@@ -404,6 +647,9 @@ export async function runTerrainCli(argv = process.argv.slice(2)) {
|
|
|
404
647
|
);
|
|
405
648
|
}
|
|
406
649
|
|
|
650
|
+
// First successful run after a failed install: clear the marker.
|
|
651
|
+
await clearInstallFailureMarker();
|
|
652
|
+
|
|
407
653
|
await runBinary(binaryPath, argv);
|
|
408
654
|
}
|
|
409
655
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "mapterrain",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.3.0",
|
|
4
4
|
"description": "Terrain test intelligence CLI.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -23,10 +23,11 @@
|
|
|
23
23
|
},
|
|
24
24
|
"scripts": {
|
|
25
25
|
"postinstall": "node bin/postinstall.js",
|
|
26
|
-
"test": "node scripts/verify-pack.js",
|
|
27
|
-
"
|
|
28
|
-
"
|
|
29
|
-
"format
|
|
26
|
+
"test": "npm run test:unit && node scripts/verify-pack.js",
|
|
27
|
+
"test:unit": "node --test scripts/test-installer-marker.mjs scripts/test-release-sanity.mjs",
|
|
28
|
+
"lint": "eslint \"bin/*.js\" \"scripts/*.js\" \"scripts/*.mjs\"",
|
|
29
|
+
"format": "prettier --write \"bin/*.js\" \"scripts/*.js\" \"scripts/*.mjs\"",
|
|
30
|
+
"format:check": "prettier --check \"bin/*.js\" \"scripts/*.js\" \"scripts/*.mjs\"",
|
|
30
31
|
"start": "node bin/terrain-cli.js",
|
|
31
32
|
"convert": "node bin/terrain-cli.js convert",
|
|
32
33
|
"lint-staged": "lint-staged",
|
|
@@ -81,4 +82,4 @@
|
|
|
81
82
|
"node": ">=22.0.0"
|
|
82
83
|
},
|
|
83
84
|
"sideEffects": false
|
|
84
|
-
}
|
|
85
|
+
}
|