@diffci.com/diffci 0.1.4 → 0.1.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +57 -14
- package/dist-client/src/client/cli.js +59 -0
- package/dist-client/src/client/verify-savings.js +193 -0
- package/docs/distribution.md +41 -2
- package/docs/npm-adoption.md +121 -0
- package/package.json +3 -1
package/README.md
CHANGED
|
@@ -4,28 +4,71 @@
|
|
|
4
4
|
[](https://docs.npmjs.com/generating-provenance-statements)
|
|
5
5
|
[](https://github.com/DiffCI/DiffCI.com)
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
dependency graph,
|
|
9
|
-
|
|
10
|
-
this repository currently implements is observe-and-compare only; nothing here can cancel, skip, or block
|
|
11
|
-
a real CI run.
|
|
7
|
+
**Find test-selection opportunities in your CI before changing what it runs.** DiffCI analyzes a
|
|
8
|
+
commit's changes and dependency graph, then reports which test files it would select, why it falls
|
|
9
|
+
back to a full run, and whether it can propose a test command. The CLI and Action are observation-only.
|
|
12
10
|
|
|
13
|
-
|
|
11
|
+
From an existing repository checkout, with Node.js 22.5+ and Git installed:
|
|
14
12
|
|
|
15
13
|
```bash
|
|
16
|
-
npx @diffci.com/diffci@latest observe
|
|
17
|
-
npx @diffci.com/diffci@latest verify-workflow
|
|
14
|
+
npx @diffci.com/diffci@latest observe --no-send
|
|
18
15
|
```
|
|
19
16
|
|
|
20
|
-
|
|
17
|
+
**Upgrade from 0.1.3:** tests excluded by a source-only `tsconfig.json` could be discovered without
|
|
18
|
+
their dependency edges, producing an incomplete selection. This is fixed in **0.1.4**. Revalidate
|
|
19
|
+
affected observations before using them as opportunity evidence; see the
|
|
20
|
+
[historical validation](docs/evidence/growth-history-01/README.md) and
|
|
21
|
+
[release qualification](docs/evidence/release-0.1.4/README.md).
|
|
22
|
+
|
|
23
|
+
The local default compares `HEAD` with its first parent; both commits must be available. For a specific
|
|
24
|
+
comparison, add `--base <base-sha> --head <head-sha>`. DiffCI prints the selection, fallback reasons,
|
|
25
|
+
and the path to a JSON report outside your checkout. `REFUSED` or `ERROR` is not a successful analysis;
|
|
26
|
+
check the reported status even when the command exits successfully. See the
|
|
27
|
+
[support matrix](docs/language-support.md) for setup requirements and supported workloads.
|
|
28
|
+
|
|
29
|
+
**Measured example:** a controlled Cal.com replay showed **44.2% net reduction in a job-equivalent
|
|
30
|
+
install + pretest + test workload**, including analysis overhead. This is one sandbox comparison,
|
|
31
|
+
not Cal.com's production savings or a prediction for your repository.
|
|
32
|
+
[Read the timings and method](docs/research/2026-08-24-calcom-execution-observability/11-frozen-identity-and-complete-job-savings.md).
|
|
33
|
+
|
|
34
|
+
Selection counts alone do not establish runtime savings. Observation mode measures neither the
|
|
35
|
+
selected test execution nor realized savings.
|
|
36
|
+
|
|
37
|
+
For a self-serve paired runtime check, run `observe` first and then run `verify-savings` against the
|
|
38
|
+
observation report. It compares your normal full command with
|
|
39
|
+
DiffCI's proposed selected command and writes JSON plus Markdown evidence; see
|
|
40
|
+
[`docs/npm-adoption.md`](docs/npm-adoption.md#self-serve-runtime-pilot).
|
|
41
|
+
|
|
42
|
+
## Observe in GitHub Actions
|
|
43
|
+
|
|
44
|
+
Save this as `.github/workflows/diffci.yml` to add a dedicated, non-blocking observation job:
|
|
21
45
|
|
|
22
46
|
```yaml
|
|
23
|
-
|
|
47
|
+
name: DiffCI observation
|
|
48
|
+
on: [push, pull_request]
|
|
49
|
+
permissions:
|
|
50
|
+
contents: read
|
|
51
|
+
jobs:
|
|
52
|
+
diffci:
|
|
53
|
+
runs-on: ubuntu-latest
|
|
54
|
+
continue-on-error: true
|
|
55
|
+
steps:
|
|
56
|
+
- uses: actions/checkout@v4
|
|
57
|
+
with:
|
|
58
|
+
fetch-depth: 0
|
|
59
|
+
- uses: DiffCI/DiffCI.com@dee4f7b938a7720d077c1124ef2ea050aa2625d6
|
|
24
60
|
```
|
|
25
61
|
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
62
|
+
Then check the workflow locally with `npx @diffci.com/diffci@latest verify-workflow`. Keep the observer
|
|
63
|
+
out of required checks and other jobs' `needs` lists. The Action adds a job summary and a
|
|
64
|
+
`diffci-observation` artifact to the run; it does not alter which tests your other jobs execute.
|
|
65
|
+
The example pins release `v0.1.4` to its full commit SHA for reproducibility.
|
|
66
|
+
|
|
67
|
+
The CLI sends no report with `--no-send`. The Action uploads a GitHub artifact by default; sending to
|
|
68
|
+
DiffCI's hosted service requires an explicitly configured endpoint and token.
|
|
69
|
+
[Installation details](docs/distribution.md) · [Seven-day pilot](docs/shadow-pilot-runbook.md)
|
|
70
|
+
|
|
71
|
+
## Project background
|
|
29
72
|
|
|
30
73
|
**This repository moved out of the [DentalPresence.in](https://github.com/adityankale190895/DentalPresence.in)
|
|
31
74
|
monorepo** (previously `diffci/` there) into its own repo on 2026-08-21, once the project outgrew being a
|
|
@@ -136,7 +179,7 @@ npm run research:sandbox:deploy
|
|
|
136
179
|
DiffCI is intended to be installable as infrastructure, not only as a hosted shadow experiment:
|
|
137
180
|
|
|
138
181
|
```yaml
|
|
139
|
-
- uses: DiffCI/DiffCI.com@
|
|
182
|
+
- uses: DiffCI/DiffCI.com@dee4f7b938a7720d077c1124ef2ea050aa2625d6
|
|
140
183
|
```
|
|
141
184
|
|
|
142
185
|
```bash
|
|
@@ -14,6 +14,7 @@
|
|
|
14
14
|
*
|
|
15
15
|
* Commands:
|
|
16
16
|
* observe analyse the checkout and write an observation report
|
|
17
|
+
* verify-savings run a paired full-versus-selected timing check
|
|
17
18
|
* verify-workflow check that a DiffCI job in this repository's workflows cannot affect other jobs
|
|
18
19
|
* version print the observer version
|
|
19
20
|
*
|
|
@@ -30,6 +31,7 @@ import { tmpdir } from "node:os";
|
|
|
30
31
|
import { dirname, join, resolve } from "node:path";
|
|
31
32
|
import { observe, isInsideRepository } from "./observe.js";
|
|
32
33
|
import { submitObservation } from "./submit.js";
|
|
34
|
+
import { formatVerifySavingsSummary, runVerifySavings, writeVerifySavingsReport } from "./verify-savings.js";
|
|
33
35
|
import { auditWorkflows, isNonInterfering } from "./workflow-guard.js";
|
|
34
36
|
function parseArgs(argv) {
|
|
35
37
|
const args = argv.slice(2);
|
|
@@ -234,16 +236,70 @@ function runVerifyWorkflow(flags, env) {
|
|
|
234
236
|
console.log("\nBlocking findings above: as written, this installation CAN change what the rest of CI does.");
|
|
235
237
|
return 1;
|
|
236
238
|
}
|
|
239
|
+
function stringFlag(flags, name) {
|
|
240
|
+
const value = flags[name];
|
|
241
|
+
return typeof value === "string" ? value : undefined;
|
|
242
|
+
}
|
|
243
|
+
function numberFlag(flags, name) {
|
|
244
|
+
const value = stringFlag(flags, name);
|
|
245
|
+
if (value === undefined)
|
|
246
|
+
return undefined;
|
|
247
|
+
const parsed = Number(value);
|
|
248
|
+
if (!Number.isFinite(parsed) || parsed < 0)
|
|
249
|
+
throw new Error(`--${name} must be a non-negative number`);
|
|
250
|
+
return parsed;
|
|
251
|
+
}
|
|
252
|
+
function parseVerifySavingsOptions(flags, env) {
|
|
253
|
+
const full = stringFlag(flags, "full");
|
|
254
|
+
const selected = stringFlag(flags, "selected");
|
|
255
|
+
const selectedFromReport = stringFlag(flags, "selected-from-report");
|
|
256
|
+
const out = stringFlag(flags, "out");
|
|
257
|
+
if (!full)
|
|
258
|
+
throw new Error("--full <command> is required");
|
|
259
|
+
if (!selected && !selectedFromReport)
|
|
260
|
+
throw new Error("--selected <command> or --selected-from-report <path> is required");
|
|
261
|
+
if (selected && selectedFromReport)
|
|
262
|
+
throw new Error("pass only one of --selected or --selected-from-report");
|
|
263
|
+
if (!out)
|
|
264
|
+
throw new Error("--out <path> is required");
|
|
265
|
+
const markdown = stringFlag(flags, "markdown");
|
|
266
|
+
return {
|
|
267
|
+
full,
|
|
268
|
+
selected,
|
|
269
|
+
selectedFromReport: selectedFromReport ? resolve(selectedFromReport) : undefined,
|
|
270
|
+
out: resolve(out),
|
|
271
|
+
markdown: markdown ? resolve(markdown) : undefined,
|
|
272
|
+
label: stringFlag(flags, "label"),
|
|
273
|
+
cwd: resolve(stringFlag(flags, "repo") ?? env.GITHUB_WORKSPACE ?? process.cwd()),
|
|
274
|
+
timeoutMs: numberFlag(flags, "timeout-ms") ?? 30 * 60 * 1000,
|
|
275
|
+
analysisOverheadMs: numberFlag(flags, "analysis-overhead-ms"),
|
|
276
|
+
tailBytes: numberFlag(flags, "tail-bytes") ?? 12_000,
|
|
277
|
+
};
|
|
278
|
+
}
|
|
279
|
+
function runVerifySavingsCommand(flags, env) {
|
|
280
|
+
const options = parseVerifySavingsOptions(flags, env);
|
|
281
|
+
const report = runVerifySavings(options);
|
|
282
|
+
writeVerifySavingsReport(report, { out: options.out, markdown: options.markdown });
|
|
283
|
+
console.log(formatVerifySavingsSummary(report));
|
|
284
|
+
console.log(` report: ${options.out}`);
|
|
285
|
+
if (options.markdown)
|
|
286
|
+
console.log(` markdown: ${options.markdown}`);
|
|
287
|
+
return report.comparison.fullCommandSucceeded && report.comparison.selectedCommandSucceeded ? 0 : 1;
|
|
288
|
+
}
|
|
237
289
|
const USAGE = `diffci - observation-only change-aware CI analysis
|
|
238
290
|
|
|
239
291
|
Usage:
|
|
240
292
|
diffci observe [--repo <path>] [--out <file>] [--base <sha> --head <sha>]
|
|
241
293
|
[--redact-paths] [--json] [--quiet] [--fail-on-error]
|
|
242
294
|
[--api-url <url> --api-token <token>] [--no-send]
|
|
295
|
+
diffci verify-savings --repo <path> --full <command>
|
|
296
|
+
(--selected <command> | --selected-from-report <file>)
|
|
297
|
+
--out <file> [--markdown <file>] [--label <name>]
|
|
243
298
|
diffci verify-workflow [--repo <path>]
|
|
244
299
|
diffci version
|
|
245
300
|
|
|
246
301
|
observe analyses the checkout and writes one JSON report. It runs nothing and changes nothing.
|
|
302
|
+
verify-savings runs both commands and reports measured paired runtime; it is an opt-in pilot command.
|
|
247
303
|
verify-workflow checks that the job running DiffCI cannot affect any other job, and exits 1 if it can.
|
|
248
304
|
|
|
249
305
|
The report is sent only when both --api-url and --api-token are given (or DIFFCI_API_URL and
|
|
@@ -262,6 +318,9 @@ async function main() {
|
|
|
262
318
|
case "observe":
|
|
263
319
|
process.exitCode = await runObserve(flags, env);
|
|
264
320
|
return;
|
|
321
|
+
case "verify-savings":
|
|
322
|
+
process.exitCode = runVerifySavingsCommand(flags, env);
|
|
323
|
+
return;
|
|
265
324
|
case "verify-workflow":
|
|
266
325
|
process.exitCode = runVerifyWorkflow(flags, env);
|
|
267
326
|
return;
|
|
@@ -0,0 +1,193 @@
|
|
|
1
|
+
import { spawnSync } from "node:child_process";
|
|
2
|
+
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
|
3
|
+
import { dirname, resolve } from "node:path";
|
|
4
|
+
function tail(value, bytes) {
|
|
5
|
+
if (Buffer.byteLength(value, "utf8") <= bytes)
|
|
6
|
+
return value;
|
|
7
|
+
return value.slice(-bytes);
|
|
8
|
+
}
|
|
9
|
+
function readSelectionFromObservation(path) {
|
|
10
|
+
const absolutePath = resolve(path);
|
|
11
|
+
const parsed = JSON.parse(readFileSync(absolutePath, "utf8"));
|
|
12
|
+
if (parsed.status !== "OBSERVED")
|
|
13
|
+
throw new Error(`--selected-from-report requires an OBSERVED report; got ${String(parsed.status)}`);
|
|
14
|
+
const command = Array.isArray(parsed.result?.proposedCommands) && typeof parsed.result.proposedCommands[0] === "string"
|
|
15
|
+
? parsed.result.proposedCommands[0]
|
|
16
|
+
: undefined;
|
|
17
|
+
if (!command)
|
|
18
|
+
throw new Error("--selected-from-report did not contain result.proposedCommands[0]");
|
|
19
|
+
const selectedTests = Array.isArray(parsed.result?.selectedTests) ? parsed.result.selectedTests : undefined;
|
|
20
|
+
return {
|
|
21
|
+
command,
|
|
22
|
+
source: "diffci-observation",
|
|
23
|
+
observationReportPath: absolutePath,
|
|
24
|
+
selectedTestCount: selectedTests?.length,
|
|
25
|
+
totalTestCount: typeof parsed.result?.totalTestCount === "number" ? parsed.result.totalTestCount : undefined,
|
|
26
|
+
analysisOverheadMs: typeof parsed.timings?.totalMs === "number" ? parsed.timings.totalMs : undefined,
|
|
27
|
+
};
|
|
28
|
+
}
|
|
29
|
+
function resolveSelection(options) {
|
|
30
|
+
if (options.selectedFromReport)
|
|
31
|
+
return readSelectionFromObservation(options.selectedFromReport);
|
|
32
|
+
if (!options.selected)
|
|
33
|
+
throw new Error("--selected <command> or --selected-from-report <path> is required");
|
|
34
|
+
return { command: options.selected, source: "manual" };
|
|
35
|
+
}
|
|
36
|
+
function measureCommand(command, options) {
|
|
37
|
+
const startedAt = new Date().toISOString();
|
|
38
|
+
const started = Date.now();
|
|
39
|
+
const shellCommand = process.platform === "win32" ? "powershell.exe" : "sh";
|
|
40
|
+
const shellArgv = process.platform === "win32" ? ["-NoProfile", "-NonInteractive", "-Command", command] : ["-c", command];
|
|
41
|
+
const result = spawnSync(shellCommand, shellArgv, {
|
|
42
|
+
cwd: options.cwd,
|
|
43
|
+
encoding: "utf8",
|
|
44
|
+
timeout: options.timeoutMs,
|
|
45
|
+
maxBuffer: 256 * 1024 * 1024,
|
|
46
|
+
shell: false,
|
|
47
|
+
env: {
|
|
48
|
+
...process.env,
|
|
49
|
+
CI: "1",
|
|
50
|
+
FORCE_COLOR: "0",
|
|
51
|
+
COREPACK_ENABLE_DOWNLOAD_PROMPT: "0",
|
|
52
|
+
npm_config_yes: "true",
|
|
53
|
+
},
|
|
54
|
+
});
|
|
55
|
+
const finishedAt = new Date().toISOString();
|
|
56
|
+
return {
|
|
57
|
+
command,
|
|
58
|
+
exitCode: result.status,
|
|
59
|
+
signal: result.signal ?? null,
|
|
60
|
+
timedOut: result.error?.code === "ETIMEDOUT",
|
|
61
|
+
startedAt,
|
|
62
|
+
finishedAt,
|
|
63
|
+
wallMs: Date.now() - started,
|
|
64
|
+
stdoutTail: tail(result.stdout ?? "", options.tailBytes),
|
|
65
|
+
stderrTail: tail(result.stderr ?? "", options.tailBytes),
|
|
66
|
+
};
|
|
67
|
+
}
|
|
68
|
+
export function buildVerifySavingsReport(input) {
|
|
69
|
+
const overhead = input.analysisOverheadMs ?? 0;
|
|
70
|
+
const netSelectedMs = input.selected.wallMs + overhead;
|
|
71
|
+
const deltaMs = input.full.wallMs - netSelectedMs;
|
|
72
|
+
const percentChange = input.full.wallMs > 0 ? (deltaMs / input.full.wallMs) * 100 : 0;
|
|
73
|
+
const fullCommandSucceeded = input.full.exitCode === 0 && !input.full.timedOut;
|
|
74
|
+
const selectedCommandSucceeded = input.selected.exitCode === 0 && !input.selected.timedOut;
|
|
75
|
+
const missedFailureSignal = !fullCommandSucceeded && selectedCommandSucceeded;
|
|
76
|
+
const notes = [
|
|
77
|
+
"This is paired runtime evidence, not a production-savings claim.",
|
|
78
|
+
"Full and selected commands were run sequentially in the same checkout.",
|
|
79
|
+
input.analysisOverheadMs === undefined
|
|
80
|
+
? "No analysis overhead was provided, so net selected runtime equals selected command runtime."
|
|
81
|
+
: "Net selected runtime includes DiffCI analysis overhead.",
|
|
82
|
+
];
|
|
83
|
+
if (missedFailureSignal)
|
|
84
|
+
notes.push("Full failed while selected passed; inspect outputs before treating the selection as safe.");
|
|
85
|
+
return {
|
|
86
|
+
schema: "diffci.verifySavings.v1",
|
|
87
|
+
producedAt: input.producedAt ?? new Date().toISOString(),
|
|
88
|
+
label: input.label,
|
|
89
|
+
cwd: input.cwd,
|
|
90
|
+
timeoutMs: input.timeoutMs,
|
|
91
|
+
analysisOverheadMs: input.analysisOverheadMs,
|
|
92
|
+
selectionSource: input.selection.source,
|
|
93
|
+
observationReportPath: input.selection.observationReportPath,
|
|
94
|
+
selectedTestCount: input.selection.selectedTestCount,
|
|
95
|
+
totalTestCount: input.selection.totalTestCount,
|
|
96
|
+
full: input.full,
|
|
97
|
+
selected: input.selected,
|
|
98
|
+
comparison: {
|
|
99
|
+
fullWallMs: input.full.wallMs,
|
|
100
|
+
selectedWallMs: input.selected.wallMs,
|
|
101
|
+
netSelectedMs,
|
|
102
|
+
deltaMs,
|
|
103
|
+
percentChange,
|
|
104
|
+
selectedCommandSucceeded,
|
|
105
|
+
fullCommandSucceeded,
|
|
106
|
+
missedFailureSignal,
|
|
107
|
+
},
|
|
108
|
+
notes,
|
|
109
|
+
};
|
|
110
|
+
}
|
|
111
|
+
function formatMs(ms) {
|
|
112
|
+
if (Math.abs(ms) < 1000)
|
|
113
|
+
return `${ms}ms`;
|
|
114
|
+
return `${(ms / 1000).toFixed(2)}s`;
|
|
115
|
+
}
|
|
116
|
+
function formatPercent(value) {
|
|
117
|
+
return `${value >= 0 ? "+" : ""}${value.toFixed(1)}%`;
|
|
118
|
+
}
|
|
119
|
+
export function renderVerifySavingsMarkdown(report) {
|
|
120
|
+
const deltaLabel = report.comparison.deltaMs >= 0 ? "faster" : "slower";
|
|
121
|
+
const overhead = report.analysisOverheadMs === undefined ? "not provided" : formatMs(report.analysisOverheadMs);
|
|
122
|
+
const title = report.label ? `# DiffCI Verify Savings: ${report.label}` : "# DiffCI Verify Savings";
|
|
123
|
+
const warning = report.comparison.missedFailureSignal
|
|
124
|
+
? "\n> WARNING: Full failed while selected passed. Do not treat this selected command as safe until the full-run failure is understood.\n"
|
|
125
|
+
: "";
|
|
126
|
+
const selectionCounts = report.selectedTestCount !== undefined && report.totalTestCount !== undefined
|
|
127
|
+
? `\nSelected tests: ${report.selectedTestCount} of ${report.totalTestCount}\n`
|
|
128
|
+
: "";
|
|
129
|
+
return `${title}
|
|
130
|
+
|
|
131
|
+
Produced at: ${report.producedAt}
|
|
132
|
+
Repository label: ${report.label ?? "not provided"}
|
|
133
|
+
Selection source: ${report.selectionSource}
|
|
134
|
+
${selectionCounts}${warning}
|
|
135
|
+
|
|
136
|
+
This report compares a full command with a selected command on the same checkout. It is measured pilot evidence, not a production-savings claim.
|
|
137
|
+
|
|
138
|
+
## Result
|
|
139
|
+
|
|
140
|
+
| Measure | Value |
|
|
141
|
+
| --- | ---: |
|
|
142
|
+
| Full runtime | ${formatMs(report.comparison.fullWallMs)} |
|
|
143
|
+
| Selected runtime | ${formatMs(report.comparison.selectedWallMs)} |
|
|
144
|
+
| DiffCI analysis overhead | ${overhead} |
|
|
145
|
+
| Net selected runtime | ${formatMs(report.comparison.netSelectedMs)} |
|
|
146
|
+
| Delta vs full | ${formatMs(report.comparison.deltaMs)} ${deltaLabel} |
|
|
147
|
+
| Percent change vs full | ${formatPercent(report.comparison.percentChange)} |
|
|
148
|
+
|
|
149
|
+
## Commands
|
|
150
|
+
|
|
151
|
+
| Arm | Exit | Timed out | Command |
|
|
152
|
+
| --- | ---: | --- | --- |
|
|
153
|
+
| Full | ${report.full.exitCode ?? "signal"} | ${report.full.timedOut ? "yes" : "no"} | \`${report.full.command.replaceAll("|", "\\|")}\` |
|
|
154
|
+
| Selected | ${report.selected.exitCode ?? "signal"} | ${report.selected.timedOut ? "yes" : "no"} | \`${report.selected.command.replaceAll("|", "\\|")}\` |
|
|
155
|
+
|
|
156
|
+
## Interpretation Notes
|
|
157
|
+
|
|
158
|
+
${report.notes.map((note) => `- ${note}`).join("\n")}
|
|
159
|
+
`;
|
|
160
|
+
}
|
|
161
|
+
function writeText(path, value) {
|
|
162
|
+
mkdirSync(dirname(path), { recursive: true });
|
|
163
|
+
writeFileSync(path, value, "utf8");
|
|
164
|
+
}
|
|
165
|
+
export function runVerifySavings(options) {
|
|
166
|
+
const selection = resolveSelection(options);
|
|
167
|
+
const analysisOverheadMs = options.analysisOverheadMs ?? selection.analysisOverheadMs;
|
|
168
|
+
const full = measureCommand(options.full, options);
|
|
169
|
+
const selected = measureCommand(selection.command, options);
|
|
170
|
+
return buildVerifySavingsReport({
|
|
171
|
+
label: options.label,
|
|
172
|
+
cwd: options.cwd,
|
|
173
|
+
timeoutMs: options.timeoutMs,
|
|
174
|
+
analysisOverheadMs,
|
|
175
|
+
selection,
|
|
176
|
+
full,
|
|
177
|
+
selected,
|
|
178
|
+
});
|
|
179
|
+
}
|
|
180
|
+
export function writeVerifySavingsReport(report, paths) {
|
|
181
|
+
writeText(paths.out, `${JSON.stringify(report, null, 2)}\n`);
|
|
182
|
+
if (paths.markdown)
|
|
183
|
+
writeText(paths.markdown, renderVerifySavingsMarkdown(report));
|
|
184
|
+
}
|
|
185
|
+
export function formatVerifySavingsSummary(report) {
|
|
186
|
+
const lines = [
|
|
187
|
+
`DiffCI verify-savings: ${report.comparison.deltaMs >= 0 ? "faster" : "slower"} by ${formatMs(Math.abs(report.comparison.deltaMs))}`,
|
|
188
|
+
];
|
|
189
|
+
if (report.comparison.missedFailureSignal) {
|
|
190
|
+
lines.push(" warning: full failed while selected passed; inspect outputs before claiming safety");
|
|
191
|
+
}
|
|
192
|
+
return lines.join("\n");
|
|
193
|
+
}
|
package/docs/distribution.md
CHANGED
|
@@ -7,6 +7,45 @@ The product model is open core. The npm CLI and basic GitHub Action are the open
|
|
|
7
7
|
DiffCI Cloud adds hosted history, organization dashboards, policies, managed operations, and support.
|
|
8
8
|
See [`open-core-packaging.md`](open-core-packaging.md).
|
|
9
9
|
|
|
10
|
+
## Channel strategy
|
|
11
|
+
|
|
12
|
+
Reviewed 2026-09-19. These channels have different jobs in the same adoption path:
|
|
13
|
+
|
|
14
|
+
| Channel | Role for DiffCI | Next step |
|
|
15
|
+
| --- | --- | --- |
|
|
16
|
+
| npm | Developer discovery and local evaluation of `@diffci.com/diffci` | Lead with `observe`, a sample report, and the observation-only contract. |
|
|
17
|
+
| GitHub Action | Repeatable CI adoption | Use the dedicated, non-blocking job below and verify the workflow before a pilot. |
|
|
18
|
+
| GitHub Marketplace | CI-specific discovery of the same Action | Confirm listing status and release requirements, then publish or improve its installation instructions. |
|
|
19
|
+
| Tidelift | Potential package maintenance assurance and maintainer income | Pursue package acceptance using the existing submission packet; do not present support as active. |
|
|
20
|
+
| Commercial DiffCI | Paid hosted history, analytics, team workflows, and managed operations | Validate demand with design partners and distinguish available services from future acceleration capabilities. |
|
|
21
|
+
|
|
22
|
+
The Action is the integration; Marketplace is a discovery channel for that integration. GitHub documents
|
|
23
|
+
Marketplace publication as a release-based process, so having `action.yml` alone does not establish
|
|
24
|
+
that a listing is live. See [GitHub's publishing requirements](https://docs.github.com/en/actions/how-tos/create-and-publish-actions/publish-in-github-marketplace).
|
|
25
|
+
|
|
26
|
+
Prioritize npm and Action activation, then Marketplace discovery. Keep Tidelift outreach in parallel
|
|
27
|
+
without making adoption dependent on acceptance or expected payments. Defer PyPI, Cargo, and Homebrew
|
|
28
|
+
until user demand and a maintained installation experience justify each additional surface.
|
|
29
|
+
|
|
30
|
+
Measure successful first reports, repositories with repeat observations, completed pilot reviews,
|
|
31
|
+
and qualified commercial interest. Downloads and listing views alone do not demonstrate adoption.
|
|
32
|
+
Use voluntary pilot feedback or explicitly configured hosted reporting for these measurements;
|
|
33
|
+
local observation does not imply permission to collect telemetry.
|
|
34
|
+
|
|
35
|
+
Suggested positioning:
|
|
36
|
+
|
|
37
|
+
> Install DiffCI through npm or GitHub Actions to inspect what your CI could avoid running while
|
|
38
|
+
> leaving CI execution unchanged. Explore hosted reports and shared history with the DiffCI team.
|
|
39
|
+
|
|
40
|
+
Acceleration and managed infrastructure remain product directions unless a specific capability is
|
|
41
|
+
available and validated. Potential savings in shadow reports are not realized customer savings.
|
|
42
|
+
|
|
43
|
+
Tidelift's published payment policy allows individuals, for-profit organizations, and nonprofits to
|
|
44
|
+
receive payments with a signed agreement. A nonprofit recipient is therefore not inherently required.
|
|
45
|
+
Payment depends on factors including subscriber usage and package importance; this does not establish
|
|
46
|
+
DiffCI's eligibility, acceptance, or income. See [Tidelift's payment policy](https://support.tidelift.com/hc/en-us/articles/4406294816916-How-we-pay-lifters)
|
|
47
|
+
and the [DiffCI submission packet](tidelift-submission.md).
|
|
48
|
+
|
|
10
49
|
## GitHub App
|
|
11
50
|
|
|
12
51
|
The DiffCI Shadow GitHub App is the lowest-friction research and design-partner path. It receives
|
|
@@ -29,10 +68,10 @@ jobs:
|
|
|
29
68
|
- uses: actions/checkout@v4
|
|
30
69
|
with:
|
|
31
70
|
fetch-depth: 0
|
|
32
|
-
- uses: DiffCI/DiffCI.com@
|
|
71
|
+
- uses: DiffCI/DiffCI.com@dee4f7b938a7720d077c1124ef2ea050aa2625d6
|
|
33
72
|
```
|
|
34
73
|
|
|
35
|
-
|
|
74
|
+
The example pins release `v0.1.4` to its full commit SHA. `npx @diffci.com/diffci@latest verify-workflow`
|
|
36
75
|
checks that the job is dedicated, read-only, not required by other jobs, and unable to alter the rest
|
|
37
76
|
of CI.
|
|
38
77
|
|
|
@@ -0,0 +1,121 @@
|
|
|
1
|
+
# Promoting the DiffCI npm Package
|
|
2
|
+
|
|
3
|
+
DiffCI's npm package should be promoted as a low-risk CI observer:
|
|
4
|
+
|
|
5
|
+
> DiffCI observes your CI and reports which tests it would have selected, without skipping, cancelling,
|
|
6
|
+
> or changing any job.
|
|
7
|
+
|
|
8
|
+
## Install
|
|
9
|
+
|
|
10
|
+
Use the npm CLI when someone wants to try DiffCI locally or inside an existing CI step:
|
|
11
|
+
|
|
12
|
+
```bash
|
|
13
|
+
npx @diffci.com/diffci@latest observe
|
|
14
|
+
npx @diffci.com/diffci@latest verify-workflow
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
Use the GitHub Action when someone wants the normal non-blocking CI installation:
|
|
18
|
+
|
|
19
|
+
```yaml
|
|
20
|
+
jobs:
|
|
21
|
+
diffci:
|
|
22
|
+
runs-on: ubuntu-latest
|
|
23
|
+
continue-on-error: true
|
|
24
|
+
permissions:
|
|
25
|
+
contents: read
|
|
26
|
+
steps:
|
|
27
|
+
- uses: actions/checkout@v4
|
|
28
|
+
with:
|
|
29
|
+
fetch-depth: 0
|
|
30
|
+
- uses: DiffCI/DiffCI.com@dee4f7b938a7720d077c1124ef2ea050aa2625d6
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
## Outreach Copy
|
|
34
|
+
|
|
35
|
+
Short version:
|
|
36
|
+
|
|
37
|
+
> I built DiffCI as an observation-only CI dependency. It looks at a PR diff and reports which tests it
|
|
38
|
+
> would have selected, but it never skips, cancels, or changes CI. You can run it with
|
|
39
|
+
> `npx @diffci.com/diffci@latest observe` or as a non-blocking GitHub Action. I am looking for OSS
|
|
40
|
+
> repos willing to run it in shadow mode for a week.
|
|
41
|
+
|
|
42
|
+
Issue/PR version:
|
|
43
|
+
|
|
44
|
+
> Would you be open to running DiffCI in shadow mode for a week? It adds one non-blocking job that
|
|
45
|
+
> observes each PR/push and writes a report artifact. It does not alter required checks, skip tests,
|
|
46
|
+
> cancel jobs, or send data anywhere unless you explicitly configure an endpoint/token.
|
|
47
|
+
|
|
48
|
+
## Pilot Ask
|
|
49
|
+
|
|
50
|
+
Ask maintainers for a small, reversible experiment:
|
|
51
|
+
|
|
52
|
+
- Run one non-blocking DiffCI job for seven days.
|
|
53
|
+
- Keep all existing CI behavior unchanged.
|
|
54
|
+
- Share the report artifacts or a summary of whether DiffCI found avoidable test work.
|
|
55
|
+
- Remove the job at any time if it is noisy, slow, or unhelpful.
|
|
56
|
+
|
|
57
|
+
## Self-Serve Runtime Pilot
|
|
58
|
+
|
|
59
|
+
Use this when a maintainer asks whether selecting fewer tests would actually make their CI faster.
|
|
60
|
+
This is a paired local measurement, not a production-savings claim.
|
|
61
|
+
|
|
62
|
+
Step 1: create an observation report without sending it anywhere.
|
|
63
|
+
|
|
64
|
+
```bash
|
|
65
|
+
npx @diffci.com/diffci@latest observe --no-send --out ./diffci-observation.json
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Step 2: run the paired pilot.
|
|
69
|
+
|
|
70
|
+
```bash
|
|
71
|
+
npx @diffci.com/diffci@latest verify-savings \
|
|
72
|
+
--label owner/repo \
|
|
73
|
+
--repo /path/to/their/repo \
|
|
74
|
+
--full "npm test" \
|
|
75
|
+
--selected-from-report /path/to/their/repo/diffci-observation.json \
|
|
76
|
+
--out ./diffci-verify-savings.json \
|
|
77
|
+
--markdown ./diffci-verify-savings.md
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
What the report means:
|
|
81
|
+
|
|
82
|
+
- Full runtime is measured from `--full`.
|
|
83
|
+
- Selected runtime is measured from DiffCI's proposed command in the observation report.
|
|
84
|
+
- DiffCI analysis overhead is imported from `timings.totalMs` in the observation report unless
|
|
85
|
+
`--analysis-overhead-ms` is provided.
|
|
86
|
+
- Net selected runtime is selected runtime plus analysis overhead.
|
|
87
|
+
- If the full command fails while the selected command passes, the report is a safety warning, not a
|
|
88
|
+
savings result.
|
|
89
|
+
|
|
90
|
+
If the proposed command needs adjustment for the repository's runner, pass the selected command
|
|
91
|
+
manually:
|
|
92
|
+
|
|
93
|
+
```bash
|
|
94
|
+
npx @diffci.com/diffci@latest verify-savings \
|
|
95
|
+
--label owner/repo \
|
|
96
|
+
--repo /path/to/their/repo \
|
|
97
|
+
--full "pnpm test" \
|
|
98
|
+
--selected "pnpm test packages/a/src/a.test.ts packages/b/src/b.test.ts" \
|
|
99
|
+
--analysis-overhead-ms 1200 \
|
|
100
|
+
--out ./diffci-verify-savings.json \
|
|
101
|
+
--markdown ./diffci-verify-savings.md
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Common command shapes:
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
--full "npm test"
|
|
108
|
+
--full "pnpm test"
|
|
109
|
+
--full "yarn test"
|
|
110
|
+
--selected "npx vitest run path/to/file.test.ts"
|
|
111
|
+
--selected "npx jest path/to/file.test.ts"
|
|
112
|
+
--selected "node --test path/to/file.test.mjs"
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
## Trust Points
|
|
116
|
+
|
|
117
|
+
- Published as `@diffci.com/diffci` on npm.
|
|
118
|
+
- Stable `latest` release is signed with npm provenance.
|
|
119
|
+
- The package boundary is checked by `npm run check:oss-boundary`.
|
|
120
|
+
- The default observer writes a local report and sends nothing without both `DIFFCI_API_URL` and
|
|
121
|
+
`DIFFCI_TOKEN`.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@diffci.com/diffci",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.5",
|
|
4
4
|
"description": "DiffCI - deterministic change-aware CI planning",
|
|
5
5
|
"license": "AGPL-3.0-only",
|
|
6
6
|
"repository": {
|
|
@@ -23,6 +23,7 @@
|
|
|
23
23
|
"dist-client/src/repo",
|
|
24
24
|
"README.md",
|
|
25
25
|
"docs/distribution.md",
|
|
26
|
+
"docs/npm-adoption.md",
|
|
26
27
|
"docs/language-support.md",
|
|
27
28
|
"SECURITY.md",
|
|
28
29
|
"SUPPORT.md",
|
|
@@ -78,6 +79,7 @@
|
|
|
78
79
|
"validation:status": "tsx scripts/diffci-validation-cli.ts status",
|
|
79
80
|
"validation:collect": "tsx scripts/diffci-validation-cli.ts collect",
|
|
80
81
|
"dogfood:compare": "tsx scripts/dogfood-compare.ts",
|
|
82
|
+
"verify:savings-pilot": "tsx scripts/verify-savings-pilot.ts",
|
|
81
83
|
"test:calibration": "tsx --test \"tests/scripts/harness-calibration.test.ts\" \"tests/scripts/compute-usage.test.ts\"",
|
|
82
84
|
"agent:equivalence": "tsx scripts/agent-equivalence.ts",
|
|
83
85
|
"dogfood:economics": "tsx scripts/dogfood-economics.ts",
|