@diffci.com/diffci 0.1.0-alpha.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +98 -0
- package/action.yml +155 -0
- package/dist-client/src/client/cli.js +281 -0
- package/dist-client/src/client/context.js +173 -0
- package/dist-client/src/client/observe.js +230 -0
- package/dist-client/src/client/report.js +63 -0
- package/dist-client/src/client/submit.js +89 -0
- package/dist-client/src/client/workflow-guard.js +300 -0
- package/dist-client/src/git/git-diff.js +379 -0
- package/dist-client/src/git/types.js +1 -0
- package/dist-client/src/planner/path-baseline.js +131 -0
- package/dist-client/src/planner/test-command.js +124 -0
- package/dist-client/src/planner/types.js +1 -0
- package/dist-client/src/repo/analyzer.js +454 -0
- package/dist-client/src/repo/graph.js +958 -0
- package/dist-client/src/repo/impact-types.js +1 -0
- package/dist-client/src/repo/impact.js +625 -0
- package/dist-client/src/repo/layout.js +75 -0
- package/dist-client/src/repo/repo-config.js +63 -0
- package/dist-client/src/repo/runner-universe.js +253 -0
- package/dist-client/src/repo/test-discovery.js +383 -0
- package/dist-client/src/repo/test-fixture-ownership.js +76 -0
- package/dist-client/src/repo/test-framework.js +117 -0
- package/dist-client/src/repo/types.js +1 -0
- package/docs/distribution.md +81 -0
- package/package.json +118 -0
package/README.md
ADDED
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
# DiffCI
|
|
2
|
+
|
|
3
|
+
DiffCI is a deterministic, change-aware CI planner: given a commit or PR, it builds a real TypeScript
|
|
4
|
+
dependency graph, computes what's actually reachable from the changed files, and proposes which CI
|
|
5
|
+
tasks/tests could safely be skipped - without ever modifying production CI behavior itself. Every mode
|
|
6
|
+
this repository currently implements is observe-and-compare only; nothing here can cancel, skip, or block
|
|
7
|
+
a real CI run.
|
|
8
|
+
|
|
9
|
+
**This repository moved out of the [DentalPresence.in](https://github.com/adityankale190895/DentalPresence.in)
|
|
10
|
+
monorepo** (previously `diffci/` there) into its own repo on 2026-08-21, once the project outgrew being a
|
|
11
|
+
subfolder. DentalPresence.in remains DiffCI's original dogfooding target - some code (the `planner`
|
|
12
|
+
DentalPresence-specific PATH baseline/task registry, a few fixture tests) still reflects that origin - but
|
|
13
|
+
the research and shadow-validation pipelines are generic and have been exercised against dozens of
|
|
14
|
+
real third-party repositories.
|
|
15
|
+
|
|
16
|
+
## Current state
|
|
17
|
+
|
|
18
|
+
Three completed research stages plus an in-progress prospective-validation stage, in order:
|
|
19
|
+
|
|
20
|
+
- **Stage 0** - a 2,000-delta historical benchmark across 20 real repositories, run through a real
|
|
21
|
+
Cloudflare orchestrator. Verdict: **GO WITH CONDITIONS**.
|
|
22
|
+
- **Stage 1A** - forensic root-cause investigation of every repository/delta where Stage 0's confidence
|
|
23
|
+
model degraded to UNSAFE, and of every historical "unsafe miss" candidate. Identified the top 3
|
|
24
|
+
highest-leverage fixes.
|
|
25
|
+
- **Stage 1B** - implemented those 3 fixes (reachability-aware confidence narrowing, an improved
|
|
26
|
+
historical safety-measurement methodology, a tsconfig-scope + package.json-diffing fix), validated them
|
|
27
|
+
live against real repositories (coverage improved, zero contradicted safety cases), and ran a real
|
|
28
|
+
wall-clock FULL/PATH/DiffCI runtime pilot.
|
|
29
|
+
- **Stage 2** (current) - prospective shadow validation on real, currently-arriving CI events, not more
|
|
30
|
+
historical benchmarking. A live pipeline (Cloudflare Sandbox Containers + Worker, D1 + R2) observes real
|
|
31
|
+
repositories, predicts *before* their outcome is known, and later reconciles against the real CI result.
|
|
32
|
+
Current verdict: **EXTEND SHADOW VALIDATION** - the pipeline is real and defect-free, and since
|
|
33
|
+
2026-08-21 it runs **autonomously**: a Cron Trigger polls enrolled repositories every 10 minutes
|
|
34
|
+
(`src/research/cloudflare/shadow-cron.ts`), and the registered **DiffCI Shadow GitHub App**
|
|
35
|
+
(read-only; see [`docs/github-app-registration.md`](docs/github-app-registration.md)) delivers
|
|
36
|
+
push/workflow events to `/v1/shadow/webhook` for instant predictions and exactly-on-time
|
|
37
|
+
reconciliation - this repository shadow-observes itself through that App. See
|
|
38
|
+
[`docs/research/2026-08-21-stage2-final-report.md`](docs/research/2026-08-21-stage2-final-report.md)
|
|
39
|
+
for the full picture; what's honestly still missing is real observation volume, working GitHub
|
|
40
|
+
Actions on our own repositories (account billing), and real design-partner repositories.
|
|
41
|
+
|
|
42
|
+
Every dated report behind these stages lives in [`docs/research/`](docs/research/) - start with
|
|
43
|
+
`2026-08-21-stage2-architecture.md` for the fullest current picture of what's built vs not, or the
|
|
44
|
+
Stage 0/1A/1B reports for the historical-validation story.
|
|
45
|
+
|
|
46
|
+
## Architecture
|
|
47
|
+
|
|
48
|
+
- `src/git/` - Git delta analysis (`analyzeGitDelta`): parses a commit range into a structured,
|
|
49
|
+
serializable `GitDelta`. Project invariant: **failure to analyze must never be interpreted as
|
|
50
|
+
permission to skip CI** - a failed analysis returns `{ success: false, error }` explicitly, never a
|
|
51
|
+
silently-empty affected set.
|
|
52
|
+
- `src/repo/` - the dependency graph engine (`buildDependencyGraph`, TypeScript-compiler-backed) and the
|
|
53
|
+
impact analyzer (`ImpactAnalyzer`) that turns a graph + delta into a confidence-scored, fallback-aware
|
|
54
|
+
impact result.
|
|
55
|
+
- `src/planner/` - turns an impact result into an `ExecutionPlan` (`DefaultCIPlanner`): which tasks/tests
|
|
56
|
+
run, which are skip-candidates, and why.
|
|
57
|
+
- `src/research/` - the Stage 0/1 historical benchmark pipeline: repository sampling, the generic
|
|
58
|
+
(non-DentalPresence-specific) PATH baseline, the opportunity classifier, historical GitHub CI evidence
|
|
59
|
+
collection with flakiness detection, and the Cloudflare orchestrator (`src/research/cloudflare/`) that
|
|
60
|
+
runs all of it at scale.
|
|
61
|
+
- `src/shadow/` - the Stage 2 prospective pipeline: event identity (`event-identity.ts`), failure
|
|
62
|
+
classification, ground-truth reconciliation (`reconcile.ts`) against real CI outcomes, and (written,
|
|
63
|
+
not yet registered) GitHub App JWT/webhook code (`github-app.ts`).
|
|
64
|
+
|
|
65
|
+
## Commands
|
|
66
|
+
|
|
67
|
+
```bash
|
|
68
|
+
# Type-check and run the full test suite
|
|
69
|
+
npm run check
|
|
70
|
+
|
|
71
|
+
# Generate an example delta / impact / plan for the current repo's latest commit
|
|
72
|
+
npm run diffci
|
|
73
|
+
npm run impact
|
|
74
|
+
|
|
75
|
+
# Run a real Stage 0-style historical benchmark locally
|
|
76
|
+
npm run research:stage0
|
|
77
|
+
|
|
78
|
+
# Deploy the Cloudflare research/shadow Worker (D1 + R2 + Sandbox Containers)
|
|
79
|
+
npm run research:sandbox:deploy
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
## Install Surfaces
|
|
83
|
+
|
|
84
|
+
DiffCI is intended to be installable as infrastructure, not only as a hosted shadow experiment:
|
|
85
|
+
|
|
86
|
+
```yaml
|
|
87
|
+
- uses: DiffCI/DiffCI.com@v1
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
npx @diffci.com/diffci observe
|
|
92
|
+
npx @diffci.com/diffci verify-workflow
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
The GitHub App remains the easiest shadow-mode entry point. The GitHub Action and npm CLI establish
|
|
96
|
+
the OSS/package distribution path: DiffCI can become an explicit CI dependency while preserving the
|
|
97
|
+
same observe-only contract. See [`docs/distribution.md`](docs/distribution.md) for the package and
|
|
98
|
+
Action positioning.
|
package/action.yml
ADDED
|
@@ -0,0 +1,155 @@
|
|
|
1
|
+
# DiffCI observer - the GitHub Action a third-party repository installs (Phase 02, 2026-08-26).
|
|
2
|
+
#
|
|
3
|
+
# WHAT THIS DOES: analyses one commit range against the checkout the job already has, writes a JSON
|
|
4
|
+
# report to the runner's temp directory, prints a summary to the job log and the job summary, optionally
|
|
5
|
+
# uploads the report as an artifact, and - only when `api-url` and `api-token` are both set - sends that
|
|
6
|
+
# same report to DiffCI. With no token configured nothing leaves the runner at all.
|
|
7
|
+
#
|
|
8
|
+
# WHAT THIS DOES NOT DO, and cannot: run tests, skip tests, cancel a job, re-order steps, write to the
|
|
9
|
+
# repository, comment on a pull request, or set a check status. There is no input below that turns any
|
|
10
|
+
# of that on, because the claim being tested for seven days is that installing DiffCI leaves CI
|
|
11
|
+
# byte-identical, and a flag that could change what CI runs would eventually be set by accident.
|
|
12
|
+
#
|
|
13
|
+
# INSTALL IT AS ITS OWN JOB. A job of its own is what makes the claim structural rather than careful:
|
|
14
|
+
#
|
|
15
|
+
# jobs:
|
|
16
|
+
# diffci:
|
|
17
|
+
# runs-on: ubuntu-latest
|
|
18
|
+
# continue-on-error: true # a DiffCI failure must not become the workflow's conclusion
|
|
19
|
+
# permissions:
|
|
20
|
+
# contents: read
|
|
21
|
+
# steps:
|
|
22
|
+
# - uses: actions/checkout@v4
|
|
23
|
+
# with:
|
|
24
|
+
# fetch-depth: 0 # the base commit must exist locally, or DiffCI refuses
|
|
25
|
+
# - uses: DiffCI/DiffCI.com@<40-character commit sha>
|
|
26
|
+
#
|
|
27
|
+
# `diffci verify-workflow` checks those properties against your own workflow files and exits non-zero
|
|
28
|
+
# if any of them does not hold. Run it before the observation window starts.
|
|
29
|
+
name: DiffCI observer
|
|
30
|
+
description: Observation-only change-aware CI analysis. Runs nothing, changes nothing, skips nothing.
|
|
31
|
+
branding:
|
|
32
|
+
icon: eye
|
|
33
|
+
color: gray-dark
|
|
34
|
+
|
|
35
|
+
inputs:
|
|
36
|
+
repository-path:
|
|
37
|
+
description: The checkout to observe. Defaults to the workspace.
|
|
38
|
+
required: false
|
|
39
|
+
default: ${{ github.workspace }}
|
|
40
|
+
base-sha:
|
|
41
|
+
description: Base commit. Leave empty to derive it from the event (pull request base, push before).
|
|
42
|
+
required: false
|
|
43
|
+
default: ""
|
|
44
|
+
head-sha:
|
|
45
|
+
description: Head commit. Must be given together with base-sha.
|
|
46
|
+
required: false
|
|
47
|
+
default: ""
|
|
48
|
+
node-version:
|
|
49
|
+
description: Node version used to run the observer. Only affects this job.
|
|
50
|
+
required: false
|
|
51
|
+
default: "22"
|
|
52
|
+
report-path:
|
|
53
|
+
description: Where to write the report. Must be outside the checkout. Defaults to RUNNER_TEMP.
|
|
54
|
+
required: false
|
|
55
|
+
default: ""
|
|
56
|
+
redact-paths:
|
|
57
|
+
description: Replace every file path in the report with a stable 12-character digest.
|
|
58
|
+
required: false
|
|
59
|
+
default: "false"
|
|
60
|
+
upload-artifact:
|
|
61
|
+
description: Upload the report as a workflow artifact.
|
|
62
|
+
required: false
|
|
63
|
+
default: "true"
|
|
64
|
+
artifact-name:
|
|
65
|
+
description: Name of the uploaded artifact.
|
|
66
|
+
required: false
|
|
67
|
+
default: diffci-observation
|
|
68
|
+
artifact-retention-days:
|
|
69
|
+
description: Retention for the uploaded artifact.
|
|
70
|
+
required: false
|
|
71
|
+
default: "7"
|
|
72
|
+
api-url:
|
|
73
|
+
description: >-
|
|
74
|
+
DiffCI ingest endpoint. Leave empty and nothing is sent anywhere - the report stays on the runner
|
|
75
|
+
as an artifact. Must be https.
|
|
76
|
+
required: false
|
|
77
|
+
default: ""
|
|
78
|
+
api-token:
|
|
79
|
+
description: >-
|
|
80
|
+
Ingest token for THIS repository, from a repository secret. Pass it as the workflow expression
|
|
81
|
+
for secrets.DIFFCI_TOKEN, never as a literal. Nothing is sent without it. (This text deliberately
|
|
82
|
+
does not spell out the expression: GitHub evaluates expressions inside action metadata, and
|
|
83
|
+
"secrets" is not a context an action file may use - a literal expression here made the whole
|
|
84
|
+
action fail to load on every run until 2026-09-06.)
|
|
85
|
+
required: false
|
|
86
|
+
default: ""
|
|
87
|
+
fail-on-error:
|
|
88
|
+
description: >-
|
|
89
|
+
Fail this step when the observation refuses or errors. Default false, so a DiffCI problem never
|
|
90
|
+
shows up as a red step in your CI. Turn it on only while debugging an installation.
|
|
91
|
+
required: false
|
|
92
|
+
default: "false"
|
|
93
|
+
|
|
94
|
+
outputs:
|
|
95
|
+
report-path:
|
|
96
|
+
description: Absolute path of the written report.
|
|
97
|
+
value: ${{ steps.observe.outputs.report-path }}
|
|
98
|
+
status:
|
|
99
|
+
description: OBSERVED, REFUSED, or ERROR.
|
|
100
|
+
value: ${{ steps.observe.outputs.status }}
|
|
101
|
+
|
|
102
|
+
runs:
|
|
103
|
+
using: composite
|
|
104
|
+
steps:
|
|
105
|
+
- name: Set up Node for the observer
|
|
106
|
+
uses: actions/setup-node@v4
|
|
107
|
+
with:
|
|
108
|
+
node-version: ${{ inputs.node-version }}
|
|
109
|
+
|
|
110
|
+
# Installed into the action's own directory, never into the observed repository: nothing here
|
|
111
|
+
# touches the workspace, its node_modules, or its lockfile. --omit=dev keeps this to TypeScript and
|
|
112
|
+
# a YAML parser; --ignore-scripts means no dependency of DiffCI's runs code on your runner.
|
|
113
|
+
- name: Install observer dependencies
|
|
114
|
+
shell: bash
|
|
115
|
+
working-directory: ${{ github.action_path }}
|
|
116
|
+
run: npm ci --omit=dev --no-audit --fund=false --ignore-scripts
|
|
117
|
+
|
|
118
|
+
- name: Build observer
|
|
119
|
+
shell: bash
|
|
120
|
+
working-directory: ${{ github.action_path }}
|
|
121
|
+
run: node_modules/.bin/tsc -p tsconfig.client.json
|
|
122
|
+
|
|
123
|
+
- name: Observe
|
|
124
|
+
id: observe
|
|
125
|
+
shell: bash
|
|
126
|
+
env:
|
|
127
|
+
DIFFCI_REPO: ${{ inputs.repository-path }}
|
|
128
|
+
DIFFCI_BASE: ${{ inputs.base-sha }}
|
|
129
|
+
DIFFCI_HEAD: ${{ inputs.head-sha }}
|
|
130
|
+
DIFFCI_OUT: ${{ inputs.report-path }}
|
|
131
|
+
DIFFCI_REDACT: ${{ inputs.redact-paths }}
|
|
132
|
+
DIFFCI_FAIL_ON_ERROR: ${{ inputs.fail-on-error }}
|
|
133
|
+
# Passed through the environment, never on the command line: an argv is visible to every other
|
|
134
|
+
# process on the runner and shows up in traces. The observer reads these directly and never
|
|
135
|
+
# prints them.
|
|
136
|
+
DIFFCI_API_URL: ${{ inputs.api-url }}
|
|
137
|
+
DIFFCI_TOKEN: ${{ inputs.api-token }}
|
|
138
|
+
run: |
|
|
139
|
+
set -euo pipefail
|
|
140
|
+
args=(observe --repo "$DIFFCI_REPO")
|
|
141
|
+
if [ -n "$DIFFCI_BASE" ]; then args+=(--base "$DIFFCI_BASE"); fi
|
|
142
|
+
if [ -n "$DIFFCI_HEAD" ]; then args+=(--head "$DIFFCI_HEAD"); fi
|
|
143
|
+
if [ -n "$DIFFCI_OUT" ]; then args+=(--out "$DIFFCI_OUT"); fi
|
|
144
|
+
if [ "$DIFFCI_REDACT" = "true" ]; then args+=(--redact-paths); fi
|
|
145
|
+
if [ "$DIFFCI_FAIL_ON_ERROR" = "true" ]; then args+=(--fail-on-error); fi
|
|
146
|
+
node "$GITHUB_ACTION_PATH/dist-client/src/client/cli.js" "${args[@]}"
|
|
147
|
+
|
|
148
|
+
- name: Upload observation report
|
|
149
|
+
if: ${{ inputs.upload-artifact == 'true' && steps.observe.outputs.report-path != '' }}
|
|
150
|
+
uses: actions/upload-artifact@v4
|
|
151
|
+
with:
|
|
152
|
+
name: ${{ inputs.artifact-name }}
|
|
153
|
+
path: ${{ steps.observe.outputs.report-path }}
|
|
154
|
+
retention-days: ${{ inputs.artifact-retention-days }}
|
|
155
|
+
if-no-files-found: warn
|
|
@@ -0,0 +1,281 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/**
|
|
3
|
+
* `diffci` - the client-side command (Phase 02, 2026-08-26).
|
|
4
|
+
*
|
|
5
|
+
* This is the binary a third-party repository runs in its own CI. Everything it does is observation:
|
|
6
|
+
* it reads a checkout, writes one JSON report to a path outside that checkout, prints a summary,
|
|
7
|
+
* optionally sends that report to DiffCI, and exits 0. There is no mode in this file that runs, skips,
|
|
8
|
+
* cancels or re-orders anything, and the absence is deliberate - the seven-day Phase 02 criterion is
|
|
9
|
+
* "CI byte-identical", and a flag that could change what CI runs is a flag that will eventually be set
|
|
10
|
+
* by accident.
|
|
11
|
+
*
|
|
12
|
+
* Sending is opt-in and off unless both an API URL and a token are supplied (Phase 03). Without them
|
|
13
|
+
* the observer is exactly what Phase 02 shipped: a local analysis whose output never leaves the runner.
|
|
14
|
+
*
|
|
15
|
+
* Commands:
|
|
16
|
+
* observe analyse the checkout and write an observation report
|
|
17
|
+
* verify-workflow check that a DiffCI job in this repository's workflows cannot affect other jobs
|
|
18
|
+
* version print the observer version
|
|
19
|
+
*
|
|
20
|
+
* Exit codes: `observe` exits 0 even when it refuses or errors, because a broken observer must not
|
|
21
|
+
* fail somebody's build; the status is in the report and in the printed summary. `--fail-on-error`
|
|
22
|
+
* opts out of that, for operators running it deliberately. The one exception is exit 2, for a
|
|
23
|
+
* misconfigured invocation that would itself break the byte-identical guarantee (a report path inside
|
|
24
|
+
* the observed checkout) - nothing was observed, and the caller has to change the call. `verify-workflow`
|
|
25
|
+
* exits 1 on a BLOCKING finding - it is a pre-install check run by a human, not a step inside a build.
|
|
26
|
+
*/
|
|
27
|
+
import { execFileSync } from "node:child_process";
|
|
28
|
+
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
|
29
|
+
import { tmpdir } from "node:os";
|
|
30
|
+
import { dirname, join, resolve } from "node:path";
|
|
31
|
+
import { observe, isInsideRepository } from "./observe.js";
|
|
32
|
+
import { submitObservation } from "./submit.js";
|
|
33
|
+
import { auditWorkflows, isNonInterfering } from "./workflow-guard.js";
|
|
34
|
+
function parseArgs(argv) {
|
|
35
|
+
const args = argv.slice(2);
|
|
36
|
+
const flags = {};
|
|
37
|
+
let command;
|
|
38
|
+
for (let i = 0; i < args.length; i++) {
|
|
39
|
+
const token = args[i];
|
|
40
|
+
if (!token.startsWith("--")) {
|
|
41
|
+
command ??= token;
|
|
42
|
+
continue;
|
|
43
|
+
}
|
|
44
|
+
const key = token.slice(2);
|
|
45
|
+
const next = args[i + 1];
|
|
46
|
+
if (next !== undefined && !next.startsWith("--")) {
|
|
47
|
+
flags[key] = next;
|
|
48
|
+
i++;
|
|
49
|
+
}
|
|
50
|
+
else {
|
|
51
|
+
flags[key] = true;
|
|
52
|
+
}
|
|
53
|
+
}
|
|
54
|
+
return { command, flags };
|
|
55
|
+
}
|
|
56
|
+
/**
|
|
57
|
+
* The observer's own version and commit. Found by walking up from this file rather than by a fixed
|
|
58
|
+
* relative path, because this module runs both from source (tsx, depth src/client) and from compiled
|
|
59
|
+
* output (node, depth dist-client/src/client) and a hardcoded `../..` is right in exactly one of them.
|
|
60
|
+
*/
|
|
61
|
+
function observerIdentity() {
|
|
62
|
+
let current = dirname(import.meta.filename);
|
|
63
|
+
for (let depth = 0; depth < 8; depth++) {
|
|
64
|
+
const candidate = join(current, "package.json");
|
|
65
|
+
if (existsSync(candidate)) {
|
|
66
|
+
try {
|
|
67
|
+
const parsed = JSON.parse(readFileSync(candidate, "utf8"));
|
|
68
|
+
if (parsed.name === "diffci") {
|
|
69
|
+
let sha;
|
|
70
|
+
try {
|
|
71
|
+
sha = execFileSync("git", ["rev-parse", "HEAD"], { cwd: current, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim();
|
|
72
|
+
}
|
|
73
|
+
catch {
|
|
74
|
+
// Installed from a tarball rather than a checkout: there is no commit to report, and
|
|
75
|
+
// inventing one would be worse than the field being absent.
|
|
76
|
+
}
|
|
77
|
+
return { version: parsed.version ?? "0.0.0", root: current, sha };
|
|
78
|
+
}
|
|
79
|
+
}
|
|
80
|
+
catch {
|
|
81
|
+
// Keep walking - an unreadable package.json above us says nothing about ours.
|
|
82
|
+
}
|
|
83
|
+
}
|
|
84
|
+
const parent = dirname(current);
|
|
85
|
+
if (parent === current)
|
|
86
|
+
break;
|
|
87
|
+
current = parent;
|
|
88
|
+
}
|
|
89
|
+
return { version: "0.0.0" };
|
|
90
|
+
}
|
|
91
|
+
function defaultReportPath(env) {
|
|
92
|
+
// RUNNER_TEMP is outside GITHUB_WORKSPACE on every GitHub-hosted runner, which is the property that
|
|
93
|
+
// matters: the report cannot become an untracked file in the repository being observed.
|
|
94
|
+
const base = env.RUNNER_TEMP && existsSync(env.RUNNER_TEMP) ? env.RUNNER_TEMP : tmpdir();
|
|
95
|
+
const stamp = env.GITHUB_RUN_ID ? `${env.GITHUB_RUN_ID}-${env.GITHUB_RUN_ATTEMPT ?? "1"}` : String(Date.now());
|
|
96
|
+
return join(base, `diffci-observation-${stamp}.json`);
|
|
97
|
+
}
|
|
98
|
+
function formatFinding(finding) {
|
|
99
|
+
const where = finding.job ? `${finding.workflow}#${finding.job}` : finding.workflow;
|
|
100
|
+
return ` [${finding.severity}] ${finding.code} (${where})\n ${finding.message}`;
|
|
101
|
+
}
|
|
102
|
+
function summarise(report) {
|
|
103
|
+
const lines = [];
|
|
104
|
+
lines.push(`DiffCI observation: ${report.status} (${report.stage})`);
|
|
105
|
+
if (report.reason)
|
|
106
|
+
lines.push(` reason: ${report.reason}`);
|
|
107
|
+
if (report.commitRange) {
|
|
108
|
+
lines.push(` range: ${report.commitRange.baseSha.slice(0, 12)}..${report.commitRange.headSha.slice(0, 12)} (${report.commitRange.source})`);
|
|
109
|
+
}
|
|
110
|
+
const result = report.result;
|
|
111
|
+
if (result) {
|
|
112
|
+
lines.push(` verdict: ${result.mode}`);
|
|
113
|
+
lines.push(` selection: ${result.selectedTests.length}/${result.totalTestCount} test files, from ${result.changedFileCount} changed file(s)`);
|
|
114
|
+
lines.push(` comparator: a simple path-rule CI would have run ${result.pathBaseline.mode === "FULL" ? "everything" : `${result.pathBaseline.selectedTestCount} test file(s)`}`);
|
|
115
|
+
lines.push(` graph: ${result.graph.nodes} nodes, confidence ${result.graph.effectiveConfidence ?? result.graph.confidence}`);
|
|
116
|
+
if (result.fallbackReasons.length > 0) {
|
|
117
|
+
lines.push(` fallback: ${result.fallbackReasons.join("; ")}`);
|
|
118
|
+
}
|
|
119
|
+
for (const command of result.proposedCommands)
|
|
120
|
+
lines.push(` would have run: ${command}`);
|
|
121
|
+
if (result.commandRefusalReason)
|
|
122
|
+
lines.push(` no command: ${result.commandRefusalReason}`);
|
|
123
|
+
if (result.blindSpot) {
|
|
124
|
+
lines.push(" blind spot: this repository declares a test framework and DiffCI discovered none of its tests");
|
|
125
|
+
}
|
|
126
|
+
}
|
|
127
|
+
lines.push(` non-interference: worktree ${report.nonInterference.worktreeUnchanged ? "unchanged" : "CHANGED - report this"}, report written ${report.nonInterference.reportWrittenOutsideRepository ? "outside" : "INSIDE"} the checkout`);
|
|
128
|
+
const blocking = report.nonInterference.workflowFindings.filter((f) => f.severity === "BLOCKING");
|
|
129
|
+
if (blocking.length > 0) {
|
|
130
|
+
lines.push(` workflow: ${blocking.length} blocking finding(s) - this installation CAN affect other jobs:`);
|
|
131
|
+
for (const finding of blocking)
|
|
132
|
+
lines.push(formatFinding(finding));
|
|
133
|
+
}
|
|
134
|
+
lines.push(" DiffCI changed nothing: no test was run, skipped, cancelled or re-ordered by this step.");
|
|
135
|
+
return lines.join("\n");
|
|
136
|
+
}
|
|
137
|
+
async function runObserve(flags, env) {
|
|
138
|
+
const repoPath = resolve(typeof flags.repo === "string" ? flags.repo : env.GITHUB_WORKSPACE ?? process.cwd());
|
|
139
|
+
const reportPath = resolve(typeof flags.out === "string" ? flags.out : defaultReportPath(env));
|
|
140
|
+
if (isInsideRepository(repoPath, reportPath) && flags["allow-in-tree-report"] !== true) {
|
|
141
|
+
// Refused rather than relocated: a caller who asked for a path inside the checkout may have a
|
|
142
|
+
// reason, and silently writing somewhere else would make the artifact they collect disappear.
|
|
143
|
+
console.error(`Refusing to write the report to ${reportPath}: it is inside the repository being observed, where an untracked file changes \`git status\` and can fail a clean-tree check. Pass --out with a path outside the checkout (RUNNER_TEMP is the default), or --allow-in-tree-report to accept that risk.`);
|
|
144
|
+
return 2;
|
|
145
|
+
}
|
|
146
|
+
const identity = observerIdentity();
|
|
147
|
+
const report = await observe({
|
|
148
|
+
repoPath,
|
|
149
|
+
env: env,
|
|
150
|
+
version: identity.version,
|
|
151
|
+
engineSha: identity.sha,
|
|
152
|
+
baseOverride: typeof flags.base === "string" ? flags.base : undefined,
|
|
153
|
+
headOverride: typeof flags.head === "string" ? flags.head : undefined,
|
|
154
|
+
redactPaths: flags["redact-paths"] === true,
|
|
155
|
+
reportPath,
|
|
156
|
+
});
|
|
157
|
+
mkdirSync(dirname(reportPath), { recursive: true });
|
|
158
|
+
writeFileSync(reportPath, `${JSON.stringify(report, null, 2)}\n`, "utf8");
|
|
159
|
+
const summary = summarise(report);
|
|
160
|
+
if (flags.json === true) {
|
|
161
|
+
console.log(JSON.stringify(report, null, 2));
|
|
162
|
+
}
|
|
163
|
+
else if (flags.quiet !== true) {
|
|
164
|
+
console.log(summary);
|
|
165
|
+
console.log(` report: ${reportPath}`);
|
|
166
|
+
}
|
|
167
|
+
// Sending happens before the job summary is written, so the summary can say whether it worked.
|
|
168
|
+
// It is allowed to fail: by this point the report is on disk and (in the action) about to become an
|
|
169
|
+
// artifact, so a delivery problem costs the observation nothing - it is DiffCI's problem to fix, not
|
|
170
|
+
// the host repository's build to fail. By this point the report is on
|
|
171
|
+
// disk and (in the action) about to become an artifact, so a delivery problem costs the observation
|
|
172
|
+
// nothing - it is DiffCI's problem to fix, not the host repository's build to fail.
|
|
173
|
+
const apiUrl = typeof flags["api-url"] === "string" ? flags["api-url"] : env.DIFFCI_API_URL;
|
|
174
|
+
const apiToken = typeof flags["api-token"] === "string" ? flags["api-token"] : env.DIFFCI_TOKEN;
|
|
175
|
+
let delivery;
|
|
176
|
+
if (apiUrl && apiToken && flags["no-send"] !== true) {
|
|
177
|
+
const outcome = await submitObservation({ apiUrl, token: apiToken, report });
|
|
178
|
+
delivery = outcome.ok
|
|
179
|
+
? `sent${outcome.duplicate ? " (already recorded - a re-run or retry of the same observation)" : ""}`
|
|
180
|
+
: `not sent (${outcome.kind}): ${outcome.message}`;
|
|
181
|
+
if (flags.quiet !== true)
|
|
182
|
+
console.log(` delivery: ${delivery}`);
|
|
183
|
+
}
|
|
184
|
+
else if (apiUrl && !apiToken && flags["no-send"] !== true) {
|
|
185
|
+
delivery = "not sent: an api-url was given with no api-token";
|
|
186
|
+
if (flags.quiet !== true)
|
|
187
|
+
console.log(` delivery: ${delivery}`);
|
|
188
|
+
}
|
|
189
|
+
// GitHub renders this under the job. It is the only place most people will ever read a report, so it
|
|
190
|
+
// says the same thing the report says, including when the answer is "refused" or "not sent".
|
|
191
|
+
if (env.GITHUB_STEP_SUMMARY) {
|
|
192
|
+
try {
|
|
193
|
+
writeFileSync(env.GITHUB_STEP_SUMMARY, `### DiffCI (observation only)\n\n\`\`\`\n${summary}${delivery ? `\n delivery: ${delivery}` : ""}\n\`\`\`\n`, { flag: "a" });
|
|
194
|
+
}
|
|
195
|
+
catch {
|
|
196
|
+
// Losing the summary is cosmetic; the report on disk is the record.
|
|
197
|
+
}
|
|
198
|
+
}
|
|
199
|
+
if (env.GITHUB_OUTPUT) {
|
|
200
|
+
try {
|
|
201
|
+
writeFileSync(env.GITHUB_OUTPUT, `report-path=${reportPath}\nstatus=${report.status}\n`, { flag: "a" });
|
|
202
|
+
}
|
|
203
|
+
catch {
|
|
204
|
+
// Same: the outputs are a convenience for the workflow, not the record.
|
|
205
|
+
}
|
|
206
|
+
}
|
|
207
|
+
if (flags["fail-on-error"] === true && report.status !== "OBSERVED")
|
|
208
|
+
return 1;
|
|
209
|
+
return 0;
|
|
210
|
+
}
|
|
211
|
+
function runVerifyWorkflow(flags, env) {
|
|
212
|
+
const repoPath = resolve(typeof flags.repo === "string" ? flags.repo : env.GITHUB_WORKSPACE ?? process.cwd());
|
|
213
|
+
const result = auditWorkflows(repoPath);
|
|
214
|
+
if (result.workflowsScanned.length === 0) {
|
|
215
|
+
console.log(`No workflow files found under ${join(repoPath, ".github", "workflows")}. Nothing was checked.`);
|
|
216
|
+
return 1;
|
|
217
|
+
}
|
|
218
|
+
console.log(`Scanned ${result.workflowsScanned.length} workflow file(s).`);
|
|
219
|
+
if (result.observerJobs.length === 0) {
|
|
220
|
+
console.log("No job runs the DiffCI action. Add one before starting the observation window.");
|
|
221
|
+
return 1;
|
|
222
|
+
}
|
|
223
|
+
console.log(`DiffCI runs in: ${result.observerJobs.join(", ")}`);
|
|
224
|
+
if (result.findings.length === 0) {
|
|
225
|
+
console.log("No findings: nothing in these workflows lets the observation change what the rest of CI does.");
|
|
226
|
+
return 0;
|
|
227
|
+
}
|
|
228
|
+
for (const finding of result.findings)
|
|
229
|
+
console.log(formatFinding(finding));
|
|
230
|
+
if (isNonInterfering(result)) {
|
|
231
|
+
console.log("\nNo blocking findings. The observation cannot change what the rest of CI does.");
|
|
232
|
+
return 0;
|
|
233
|
+
}
|
|
234
|
+
console.log("\nBlocking findings above: as written, this installation CAN change what the rest of CI does.");
|
|
235
|
+
return 1;
|
|
236
|
+
}
|
|
237
|
+
const USAGE = `diffci - observation-only change-aware CI analysis
|
|
238
|
+
|
|
239
|
+
Usage:
|
|
240
|
+
diffci observe [--repo <path>] [--out <file>] [--base <sha> --head <sha>]
|
|
241
|
+
[--redact-paths] [--json] [--quiet] [--fail-on-error]
|
|
242
|
+
[--api-url <url> --api-token <token>] [--no-send]
|
|
243
|
+
diffci verify-workflow [--repo <path>]
|
|
244
|
+
diffci version
|
|
245
|
+
|
|
246
|
+
observe analyses the checkout and writes one JSON report. It runs nothing and changes nothing.
|
|
247
|
+
verify-workflow checks that the job running DiffCI cannot affect any other job, and exits 1 if it can.
|
|
248
|
+
|
|
249
|
+
The report is sent only when both --api-url and --api-token are given (or DIFFCI_API_URL and
|
|
250
|
+
DIFFCI_TOKEN are set). A failed send is reported and never fails the step - the report is on disk
|
|
251
|
+
either way. Plain http is refused; the token is never printed.
|
|
252
|
+
`;
|
|
253
|
+
async function main() {
|
|
254
|
+
const { command, flags } = parseArgs(process.argv);
|
|
255
|
+
const env = process.env;
|
|
256
|
+
if (flags.help === true || command === "help" || command === undefined) {
|
|
257
|
+
console.log(USAGE);
|
|
258
|
+
process.exitCode = command === undefined ? 1 : 0;
|
|
259
|
+
return;
|
|
260
|
+
}
|
|
261
|
+
switch (command) {
|
|
262
|
+
case "observe":
|
|
263
|
+
process.exitCode = await runObserve(flags, env);
|
|
264
|
+
return;
|
|
265
|
+
case "verify-workflow":
|
|
266
|
+
process.exitCode = runVerifyWorkflow(flags, env);
|
|
267
|
+
return;
|
|
268
|
+
case "version":
|
|
269
|
+
console.log(observerIdentity().version);
|
|
270
|
+
return;
|
|
271
|
+
default:
|
|
272
|
+
console.error(`Unknown command "${command}".\n\n${USAGE}`);
|
|
273
|
+
process.exitCode = 1;
|
|
274
|
+
}
|
|
275
|
+
}
|
|
276
|
+
main().catch((error) => {
|
|
277
|
+
// Reaching here means a defect outside observe()'s own guard. It still must not take a build down:
|
|
278
|
+
// the failure is printed, and the exit code stays 0 unless the caller asked otherwise.
|
|
279
|
+
console.error(`DiffCI observer failed: ${error instanceof Error ? error.message : String(error)}`);
|
|
280
|
+
process.exitCode = process.argv.includes("--fail-on-error") ? 1 : 0;
|
|
281
|
+
});
|