jules-orchestrator-kit 0.66.0 → 0.68.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +135 -0
- package/CHANGELOG.md +824 -0
- package/README.md +4 -3
- package/ROADMAP_V1.md +202 -0
- package/bin/agentctl.mjs +20 -1
- package/package.json +5 -2
- package/scripts/doc-sync-check.mjs +9 -0
- package/scripts/guard-reach-check.mjs +24 -0
- package/scripts/run-tests.mjs +18 -1
- package/src/engine.mjs +39 -2
- package/src/guard-policy.mjs +151 -0
- package/src/security.mjs +134 -38
- package/src/stack-detector.mjs +46 -1
package/README.md
CHANGED
|
@@ -68,7 +68,8 @@ git add .agent AGENTS.md SPEC.md CONSTRAINTS.md .gitignore && git commit -m "cho
|
|
|
68
68
|
|
|
69
69
|
```bash
|
|
70
70
|
# 3. Author a scoped, verified task envelope with guardrails & secret scrubbing
|
|
71
|
-
|
|
71
|
+
# Interactive by default. Pass the prompt to skip straight to review:
|
|
72
|
+
npx jules-orchestrator-kit task create -p "Refactor the invoice module"
|
|
72
73
|
```
|
|
73
74
|
|
|
74
75
|
`init` reads the repository, not a template: it detects the stack, picks a
|
|
@@ -207,7 +208,7 @@ To maximize PR merge rates, dispatch tasks according to deterministic boundaries
|
|
|
207
208
|
* **Fail-Closed Security & Secret Redaction:** Evaluates explicit Deny rules before Allow rules against canonicalized, case-folded paths. Redacts high-entropy keys and base64-encoded credentials (such as Kubernetes `Secret` manifests).
|
|
208
209
|
* **Complexity & Cost Router:** Zero-dependency heuristic classifier (`src/router.mjs`) routing mechanical tasks to lightweight models while reserving primary models for complex refactors, with a `node --check` syntax-verification gate that transparently escalates a FAST-tier result to the primary provider if it left broken JS on disk.
|
|
209
210
|
* **Terminal UI & Diagnostic Matrix (`agentctl doctor`):** Interactive terminal dashboard, task sidecar manager, and automated transactional self-repair.
|
|
210
|
-
* **Verified Test Suite:** Tested with **
|
|
211
|
+
* **Verified Test Suite:** Tested with **1163 unit tests across 166 suites passing in < 15.0s**.
|
|
211
212
|
|
|
212
213
|
<br/>
|
|
213
214
|
|
|
@@ -240,7 +241,7 @@ To maximize PR merge rates, dispatch tasks according to deterministic boundaries
|
|
|
240
241
|
| `doctor` | `agentctl doctor [--probe] [--json]` | Diagnostic check runner. `--probe` additionally starts the configured provider's CLI to confirm it answers, rather than only finding it on `PATH`. | `0` (Healthy), `1` (Failures) |
|
|
241
242
|
| `queue` | `agentctl queue [--dag] [--concurrency <n>] [--dry-run] [--json]` | Consumes and executes task envelopes in `.agent/jules-queue/` with Kahn's DAG dependency resolution. Non-task files (manifests, `README.md`) are skipped, and `--dry-run` previews without moving anything. | `0` (Complete) |
|
|
242
243
|
| `swarm` | `agentctl swarm [--json]` | Runs parallel multi-agent swarm across worker slots with PID liveness detection. | `0` (Complete) |
|
|
243
|
-
| `check` / `gate` / `audit`| `agentctl check [--mode working-tree] [--fix] [--allow-protected] [--allow-test-change <kind>] [--json] [--json-report <path>]` | Runs security, secret scanning, rules budget audit, and tiered verification gates (with declarative assertion support) against working tree or branch. | `0` (Approved), `1` (Budget/Arg), `3` (Scope), `4` (Verify), `5` (Diff >75K), `6` (Secret), `8` (Flaky) |
|
|
244
|
+
| `check` / `gate` / `audit`| `agentctl check [--mode working-tree] [--fix] [--allow-protected] [--allow-test-change <kind>] [--json] [--json-report <path>]` | Runs security, secret scanning, rules budget audit, and tiered verification gates (with declarative assertion support) against working tree or branch. | `0` (Approved), `1` (Budget/Arg), `3` (Scope), `4` (Verify), `5` (Diff >75K), `6` (Secret **or** test integrity), `8` (Flaky) |
|
|
244
245
|
| `mutate` / `mutation` | `agentctl mutate [--min-score <n>] [--max-mutants <n>] [--cmd <testCmd>] [--json]` | Runs zero-dependency diff mutation testing harness on changed hunks with operator inversion and safety rollback. | `0` (Passed), `1` (Score Low) |
|
|
245
246
|
| `coverage` | `agentctl coverage [--min <pct>] [--cmd <testCmd>] [--base <ref>] [--json]` | Runs native zero-dependency V8 diff coverage check against added diff lines. | `0` (Passed), `1` (Low Coverage) |
|
|
246
247
|
| `probe` / `stability` | `agentctl probe [--repeat <n>] [--min <passRate>] [--cmd <testCmd>] [--json]` | Probes test suite flakiness across N consecutive iterations with oscillation detection. | `0` (Passed), `1` (Flaky) |
|
package/ROADMAP_V1.md
ADDED
|
@@ -0,0 +1,202 @@
|
|
|
1
|
+
# 🗺️ Jules Orchestrator Kit — Roadmap to v1.0 & Beyond
|
|
2
|
+
|
|
3
|
+
The **jules-orchestrator-kit** is the zero-dependency safety gatekeeper and self-healing engineering kernel for autonomous coding agents — **Google Jules**, **Claude Code**, **Codex** and the **Gemini CLI** — in any repository and any stack.
|
|
4
|
+
|
|
5
|
+
> [!IMPORTANT]
|
|
6
|
+
> **Zero Runtime Dependencies is a strict core invariant.**
|
|
7
|
+
> Every feature on this roadmap is built strictly with native Node.js 20+ built-in modules (`node:fs`, `node:child_process`, `node:crypto`, `node:http`, `node:readline`, `node:test`).
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## 📌 Release Milestones Overview
|
|
12
|
+
|
|
13
|
+
```
|
|
14
|
+
v0.68.0 (Current Stable) ──► v0.69.0 (Distributed Swarms & Leases) ──► v1.0.0 (Production Hardened Kernel)
|
|
15
|
+
(Can It Still Go Red?) (Multi-Agent DAG & Resource Locks) (Enterprise Telemetry & SLA)
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
---
|
|
19
|
+
|
|
20
|
+
## ✅ Shipped Milestones (v0.20.0 – v0.68.0)
|
|
21
|
+
|
|
22
|
+
|
|
23
|
+
### v0.68.0: Not An Approval Either
|
|
24
|
+
- [x] **An Unreadable Dialect Blocks (`src/security.mjs`)** — the guard said it had not checked, and approved anyway.
|
|
25
|
+
- [x] **A Command That Cannot Fail Is Not An Oracle (`src/engine.mjs`)** — `task create` already refused what the gate accepted.
|
|
26
|
+
- [x] **chai And node-tap (`src/security.mjs`)** — a dot chain where RSpec has a space, and a receiver named `ct`.
|
|
27
|
+
- [x] **A Reformat Is Not A Weakening (`src/security.mjs`)** — the weakening count was still line-based.
|
|
28
|
+
- [x] **`bootstrap` Declines (`src/stack-detector.mjs`)** — rather than writing an oracle that asserts its own impossibility.
|
|
29
|
+
|
|
30
|
+
### v0.67.0: A Move Is Not A Deletion
|
|
31
|
+
- [x] **In-Body Skips (`src/security.mjs`)** — a test could be silenced with `self.skipTest()`, the form unittest's own documentation uses.
|
|
32
|
+
- [x] **Cross-File Moves (`src/security.mjs`)** — refactoring produced CRITICAL findings for assertions that still run.
|
|
33
|
+
- [x] **The Published Package Explains Itself (`scripts/run-tests.mjs`)** — `npm test` crashed with a raw ENOENT in an installed copy.
|
|
34
|
+
- [x] **The Label Names What Failed (`bin/agentctl.mjs`)** — a deleted assertion was reported under a bare SECRETS heading.
|
|
35
|
+
|
|
36
|
+
### v0.66.0: Saying Nothing Is Not Saying Approved
|
|
37
|
+
- [x] **An Assertion Is A Statement (`src/security.mjs`)** — a rewritten expected value whose keyword sat on a context line collected five green phases.
|
|
38
|
+
- [x] **Line Comments Are Stripped (`src/security.mjs`)** — they never were; `pending` was left at the comment and the copy after the loop put it back.
|
|
39
|
+
- [x] **The Boundary Reaches The Operator (`src/security.mjs`)** — the warning was wired to a function the gate does not call.
|
|
40
|
+
- [x] **Evidence Before Rules (`src/memory.mjs`)** — a hardcoded sentence was being injected into every prompt as if it were a fix.
|
|
41
|
+
|
|
42
|
+
### v0.65.0: The First Hour
|
|
43
|
+
- [x] **The Quickstart No Longer Rejects Its Own Output (`src/engine.mjs`)** — `init` wrote the scaffold, told the user to commit it, and the gate called that commit a scope violation while advising `--allow-protected`.
|
|
44
|
+
- [x] **A Stated Count Outranks A Phrase (`src/ops/test-collection.mjs`)** — a healthy 190-test TAP suite was rejected as empty because a skipped fixture printed `no tests found`.
|
|
45
|
+
- [x] **`init` Probes On The Headless Path (`src/wizard-init.mjs`)** — `--yes` means "do not ask me", not "do not check".
|
|
46
|
+
- [x] **`verify.minTests` Reaches The Floor (`src/engine.mjs`)** — the failure message named a lever that was connected to nothing.
|
|
47
|
+
|
|
48
|
+
### v0.64.0: What The Guard Could Not Read
|
|
49
|
+
- [x] **The Package Is Checked The Way It Is Installed (`scripts/package-integrity-check.mjs`)** — the tarball's own import graph, resolved inside the tarball. The previous release shipped the activation-coverage check without the contract it imports.
|
|
50
|
+
- [x] **Five Silent Ecosystems (`src/security.mjs`)** — JUnit, RSpec, PHPUnit, Minitest and XCTest all returned `PASS` on a rewritten expected value, and `assertEqual` had been working only by accident.
|
|
51
|
+
- [x] **`UNREADABLE` (`src/security.mjs`)** — coverage ending is fine; coverage ending silently is not.
|
|
52
|
+
- [x] **Contracts For The Opposite Failure (`src/guard-policy.mjs`)** — ten ordinary edits that must produce no finding, because a guard that flags everything is switched off within a week.
|
|
53
|
+
|
|
54
|
+
### v0.63.0: Can It Still Go Red?
|
|
55
|
+
- [x] **Activation Coverage (`scripts/guard-reach-check.mjs`)** — Blocking in CI and in the release. Eleven known-bad canaries across six ecosystems must each still produce the finding they name; four hand-written mutants of the applicability predicate must each break at least one. Runs in 88 ms.
|
|
56
|
+
- [x] **A Hand-Written Policy Contract (`src/guard-policy.mjs`)** — Derived from what the tool advertises, never from the regexes that implement it, so the implementation cannot be its own oracle.
|
|
57
|
+
- [x] **Verdicts Carry Their Denominator (`src/security.mjs`)** — `checkTestTampering` reports `inputsSeen` and a `PASS`/`FAIL`/`NOT_APPLICABLE` status, so "checked and clean" is no longer byte-identical to "nothing was checked".
|
|
58
|
+
|
|
59
|
+
|
|
60
|
+
### v0.62.0: No Denominator, No Claim
|
|
61
|
+
- [x] **A File Is Not A Claim (`src/stack-detector.mjs`)** — `make test` requires the Makefile to declare the target; `app.json` requires a package.json beside it; a `test` script that is `echo … && exit 0` is a placeholder, not an oracle.
|
|
62
|
+
- [x] **The Generated Fallback Oracle Can Fail (`src/stack-detector.mjs`)** — It checks that every source file parses instead of asserting that the working directory exists, and refuses to pass with nothing to check.
|
|
63
|
+
- [x] **Lockfiles And Toolchain Pins Are Protected (`src/config.mjs`)** — `package.json` was protected and `package-lock.json` was not. Plus `.envrc`, `.git-credentials`, cloud credential trees and CodeBuild.
|
|
64
|
+
- [x] **The Sixth Spelling Of "Where Are The Tests" (`src/evidence.mjs`)** — Go co-located tests and every monorepo produced `fileCount: 0`, which switched `strictTestLock` off in silence.
|
|
65
|
+
- [x] **Seven More Runners In The Collection Floor** — Maven, Gradle, PHPUnit, RSpec, dotnet, XCTest, ctest, ExUnit.
|
|
66
|
+
- [x] **100% Of Nothing Is Not 100% (`src/coverage.mjs`)** — Diff coverage reports `scored: false` where V8 measured nothing, instead of the best possible number.
|
|
67
|
+
|
|
68
|
+
|
|
69
|
+
### v0.61.0: Green Against Nothing
|
|
70
|
+
- [x] **Collection Floor (`src/ops/test-collection.mjs`)** — A verification command that exits 0 having collected zero tests is no longer a pass. The count is read from the runner's own summary across node:test, pytest, cargo, jest, vitest, mocha and go; an unrecognised runner states no count and is not failed for it.
|
|
71
|
+
|
|
72
|
+
|
|
73
|
+
### v0.60.0: A Guard Worth Reading
|
|
74
|
+
- [x] **Per-Check Overrides (`--allow-test-change <kind>`)** — Answering one tamper finding no longer silences the five other checks in the bundle. `--allow-test-modifications` remains as the blunt form.
|
|
75
|
+
- [x] **Unchanged Assertions Cancel Before Pairing (`src/security.mjs`)** — Reordering or moving an assertion is no longer reported as two rewritten expectations.
|
|
76
|
+
- [x] **A Reworded Message Is Not A Rewritten Expectation (`src/security.mjs`)** — Message-position string arguments are compared apart from value positions, so improving the wording of a failure is silent while `assert.equal(name(), "Alice")` → `"Bob"` still fires.
|
|
77
|
+
|
|
78
|
+
|
|
79
|
+
### v0.59.0: One Answer Per Question
|
|
80
|
+
- [x] **One Test-Path Classifier (`src/test-paths.mjs`)** — Five modules had five definitions of "is this a test file?"; the tamper guard's could not see `tests/test_calc.py`, so every check in it was off for the standard pytest, Rust and RSpec layouts. All five now share `isTestPath`.
|
|
81
|
+
- [x] **Statement-Level Expectation Pairing (`src/security.mjs`)** — Assertions are reassembled from the diff's two images before pairing, so wrapping one across lines no longer walks through the guard. Contributed as PR #14 by an external agent; verified and merged.
|
|
82
|
+
- [x] **Numeric Literals Beyond Decimal (`src/security.mjs`)** — Hex, binary, octal, exponent and underscored literals normalise, so `0xFF` → `0xFE` is a rewritten expectation like any other.
|
|
83
|
+
|
|
84
|
+
|
|
85
|
+
### v0.58.0: Locks That Lock
|
|
86
|
+
- [x] **A CLI Lock Is A Lease (`src/state.mjs`, `bin/agentctl.mjs`)** — `lock acquire` no longer witnesses its own exiting process, so two agents can no longer both hold the same file. `--ttl` bounds it; `--pid` binds it to a real long-lived process instead.
|
|
87
|
+
- [x] **A URL No Longer Disables The Secret Scanner (`src/security.mjs`)** — URLs, data: URIs and integrity hashes are stripped from a line rather than skipping the whole line, and a URL's userinfo and query values are scanned on their own.
|
|
88
|
+
- [x] **Untracked Symlinks Are Judged By Their Target (`src/git.mjs`)** — Working-tree mode resolves links git has never seen, and a symlink is rendered as its target *path* rather than read through.
|
|
89
|
+
- [x] **Rewritten Expectations Are Reported (`src/security.mjs`)** — An assertion whose literal changed while its shape did not is `ASSERTION_EXPECTATION_CHANGED`, with `--allow-test-modifications` as the documented override.
|
|
90
|
+
- [x] **Scope Guard Covers Every Forge (`src/config.mjs`)** — GitLab, CircleCI, Jenkins, Azure, Travis, Drone, Buildkite and Woodpecker definitions are denied alongside `.github/**`; build and test-runner configuration is protected.
|
|
91
|
+
- [x] **Python Runs As A Module (`src/stack-detector.mjs`)** — `python -m pytest` under whichever interpreter name this machine has, so an ordinary layout is not rejected on first contact.
|
|
92
|
+
- [x] **Build Artifacts Do Not Read As Tampering (`src/evidence.mjs`)** — Caches a test run writes itself are excluded from the integrity hash.
|
|
93
|
+
|
|
94
|
+
|
|
95
|
+
### v0.57.0: Honest Reporting
|
|
96
|
+
- [x] **Stack-Native TDD Oracles (`src/ops/tdd-generator.mjs`)** — Generated in the runner's own language, with a RED check that requires the assertion to have actually run rather than accepting any non-zero exit.
|
|
97
|
+
- [x] **JSON Envelopes Reach The DAG Runner (`src/dag-engine.mjs`)** — `isDagTaskFile` is exported and used for the CLI's count, so a queue of `.json` envelopes is no longer reported as empty.
|
|
98
|
+
- [x] **`swarm` Honours Its Flags (`bin/agentctl.mjs`)** — `--json`, `--dry-run` and `--concurrency` are parsed, and the registry describes a dispatcher rather than an inspector.
|
|
99
|
+
- [x] **No Score Where Nothing Was Measured (`src/mutation.mjs`)** — An empty mutant population reports `null` and a reason instead of a vacuous 100%.
|
|
100
|
+
- [x] **New Files Are Visible To Every Gate (`src/git.mjs`)** — The synthetic diff for an untracked file carries a `@@` header, so mutation and diff coverage can place its lines at all.
|
|
101
|
+
|
|
102
|
+
|
|
103
|
+
### v0.56.0: Wired, Not Just Shipped
|
|
104
|
+
- [x] **File-Aware Locking (`src/state.mjs`)** — `acquireLock()` compares the requested paths against every live lock instead of only the task id.
|
|
105
|
+
- [x] **Checkpoints That Exist (`src/engine.mjs`, `src/session-ops.mjs`)** — Taken before a dispatch and before `patch --apply`, so `agentctl rollback` finally has something to restore.
|
|
106
|
+
- [x] **Reachable Monorepo Scoping (`verify.scope: affected`)** — The boundary resolver is called by the gate, opt-in, defaulted on for repositories detected as monorepos and widening again when a shared file changes.
|
|
107
|
+
- [x] **Process-Group Reaping For The Test Runner (`scripts/run-tests.mjs`)** — An interrupted run takes its whole tree with it instead of orphaning every process below the first level. A first instance of the v0.60.0 guillotine, applied where the leak was actually observed.
|
|
108
|
+
- [x] **Live-Path Corrections (`src/provider.mjs`, `src/engine.mjs`)** — `listSources()` works against the real API, and Node's test-runner context no longer masks a failing verification command.
|
|
109
|
+
|
|
110
|
+
|
|
111
|
+
### v0.55.0: Verification Integrity
|
|
112
|
+
- [x] **Assertion Weakening Detection (`src/security.mjs`)** — A specific expectation swapped for a vague one is reported as `ASSERTION_WEAKENED`, closing the one-out-one-in bypass that counting assertions could never see.
|
|
113
|
+
- [x] **Symlink-Aware Scope (`src/git.mjs`, `src/engine.mjs`)** — Added links are judged by the path they resolve to as well as their own name, without following anything on disk.
|
|
114
|
+
- [x] **Source-Bound Evidence (`src/evidence.mjs`)** — Manifests attest to the source tree, not only the tests, and the integrity hash finally sees test files that live at the repository root.
|
|
115
|
+
|
|
116
|
+
|
|
117
|
+
### v0.54.1: The Gate Fails Closed
|
|
118
|
+
- [x] **No Oracle, No Approval (`src/engine.mjs`)** — A change that ran zero verification commands is rejected rather than approved; `verify.required: false` is the deliberate opt-out, read from the base commit.
|
|
119
|
+
- [x] **Binary Payload Inspection (`src/security.mjs`, `src/git.mjs`)** — Files git renders as "Binary files ... differ" are read directly for structured credentials, closing the NUL-byte bypass, and their real size is charged against the payload ceiling.
|
|
120
|
+
|
|
121
|
+
|
|
122
|
+
### v0.54.0: First-Run Friction — Honest Errors, Detected Defaults
|
|
123
|
+
- [x] **Base Branch Detection (`src/git.mjs`)** — `detectDefaultBranch()` resolves `origin/HEAD`, a local `main`/`master`, then the checked-out branch, replacing the hardcoded `main` that failed the first gate in every `master` or `develop` repository.
|
|
124
|
+
- [x] **Diagnosable Gate Output (`bin/agentctl.mjs`, `src/ops/verify-output.mjs`)** — Phase errors are rendered instead of swallowed, and a failing stage's stdout is no longer discarded because the spawn wrapper wrote one line to stderr.
|
|
125
|
+
- [x] **Truthful Remediation Hints (`bin/agentctl.mjs`)** — Test tampering no longer advises rotating API keys, and a provider that refused the dispatch is no longer reported as an exhausted repair loop.
|
|
126
|
+
- [x] **Provider-First Onboarding & `agentctl provider set` (`src/wizard-init.mjs`, `src/config-edit.mjs`)** — The wizard asks which agent before asking about one vendor's plans, and switching providers no longer means re-running onboarding.
|
|
127
|
+
- [x] **Named Targets And Flags Actually Take Effect (`bin/agentctl.mjs`, `src/ops/ide-scaffold.mjs`, `src/ops/cli-intent.mjs`)** — `mcp init <target>`, `task create --title/--prompt` and `doctor --probe` all did something other than what they said; each now does what it says, or is no longer advertised.
|
|
128
|
+
- [x] **Onboarding Trap Distinguished From Overreach (`src/git.mjs`)** — `partitionTracked()` tells uncommitted scaffolding apart from an agent editing its own rules, so the first-run exit 3 advises committing rather than bypassing the gate.
|
|
129
|
+
- [x] **Rehearsals Write Nothing (`src/wizard-task.mjs`)** — `task create --dry-run` synthesizes and validates the envelope without queueing it.
|
|
130
|
+
|
|
131
|
+
|
|
132
|
+
### v0.53.0: Universal Portability — Any Stack, Any Agent, Any CI
|
|
133
|
+
- [x] **Provider Readiness Probe (`src/provider-readiness.mjs`, `agentctl providers`)** — Per-provider capability descriptors: a credential for the hosted `jules` adapter, a `PATH` binary for the `claude-code`/`codex`/`gemini-flash` exec adapters, with a cross-platform PATHEXT-aware resolver that spawns nothing.
|
|
134
|
+
- [x] **Vendor-Neutral Environment Spellings (`src/env-aliases.mjs`)** — Every `JULES_*` knob also answers to `AGENT_*`, normalised once at CLI entry, with the legacy name always winning.
|
|
135
|
+
- [x] **Verification Profiles (`src/profiles.mjs`, `agentctl profile`)** — `minimal | standard | max` expanded at load time into a stack-aware pipeline that skips unsupported gates with a stated reason rather than failing the diff.
|
|
136
|
+
- [x] **Generated Stack-Aware CI (`src/ci-templates.mjs`, `agentctl ci init`)** — GitHub Actions and GitLab jobs carrying the detected stack's toolchain, replacing the copy of this repository's own nine-way Node matrix.
|
|
137
|
+
- [x] **Consumer Repository Hygiene (`bin/init.js`)** — `init` no longer copies the kit's twenty internal scripts into the target repository, and both entry points now write the same manifest pair.
|
|
138
|
+
|
|
139
|
+
|
|
140
|
+
### v0.52.x: Entropy Hardening, Egress Auditing & CI Defense
|
|
141
|
+
- [x] **Binary Asset Classification Guard (`src/security.mjs`)** — Restricts binary asset skipping to tokens $\ge 256$ chars, preventing length $\pmod 4 = 0$ keys from bypassing entropy checks.
|
|
142
|
+
- [x] **CI Runtime & Egress Security (`.github/workflows/*.yml`)** — Linux eBPF runtime monitoring and network egress auditing via StepSecurity.
|
|
143
|
+
- [x] **GitHub Actions Supply Chain Hardening (`.github/`)** — Action SHA pinning, least-privilege `contents: read`, and automated Zizmor CI gate.
|
|
144
|
+
- [x] **Assertion Removal & Anti-Tamper Guard (`src/security.mjs`)** — Detects test deletion bypass (`ASSERTION_REMOVAL`) and vacuous assertion weakening.
|
|
145
|
+
- [x] **Shannon Entropy Diff Scanner (`src/security.mjs`)** — High-entropy token detection (>4.5 entropy, $\ge 24$ chars) on added diff lines.
|
|
146
|
+
- [x] **Jules API Patch Ingestion & Edge Webhooks (`src/session-ops.mjs`, `src/webhook.mjs`)** — API changeset extraction and Webhook `Uint8Array` Edge runtime parity.
|
|
147
|
+
- [x] **Zero-Test Oracle Bootstrapping & Git Remote Origin Auto-Detection (`src/stack-detector.mjs`, `src/git.mjs`)** — Auto-configuration for virgin repositories.
|
|
148
|
+
|
|
149
|
+
### v0.51.0: Mechanical Falsification, AST Mutation & V8 Coverage
|
|
150
|
+
- [x] **Diff-Hunk Mutation Testing Engine (`src/mutation.mjs`, `agentctl mutate`)** — Transactional operator inversion with automatic shadow disk rollback.
|
|
151
|
+
- [x] **Native V8 Diff Coverage Enforcer (`src/coverage.mjs`, `agentctl coverage`)** — Zero-dependency V8 coverage mapping against added `+` hunks.
|
|
152
|
+
- [x] **Test-Oscillation Cycle Detector (`src/remediation.mjs`, `src/engine.mjs`)** — Halts multi-test oscillation cycles in OODA repair loops.
|
|
153
|
+
- [x] **Flakiness Stability Prober & Event Loop Delay Guard (`src/stability.mjs`, `src/perf.mjs`)** — Multi-pass repetition probing and Big-O event loop delay monitor.
|
|
154
|
+
|
|
155
|
+
### v0.40.0 – v0.50.0: Architecture Governance & Universal Envelopes
|
|
156
|
+
- [x] **Universal Stack-Agnostic Task Templates (`src/web-templates.mjs`)** — Pre-calibrated templates for Cargo, Go, Python, PHP, .NET, Java, and Node.
|
|
157
|
+
- [x] **Specialist Agent Roles (`.agent/prompts/`)** — Eight stack-neutral specialist personas with dynamic command hydration.
|
|
158
|
+
- [x] **All-In-One CI Verification Gate (`agentctl check`)** — Unified security, scope, payload, and stack test gatekeeper.
|
|
159
|
+
- [x] **Stack-Tailored Contract Scaffolding (`src/scaffold.mjs`)** — Automatic `SPEC.md`, `CONSTRAINTS.md`, and `DESIGN.md` generation on init.
|
|
160
|
+
|
|
161
|
+
### v0.30.0 – v0.39.0: Budget Provenance, Terminal UX & Autonomous Resilience
|
|
162
|
+
- [x] **Rolling 24h Budget Ledger & Limit Provenance (`src/budget.mjs`, `agentctl budget`)** — Multi-user attribution, rolling window accounting, and graceful estimate warnings.
|
|
163
|
+
- [x] **Terminal UI Engine (`src/tui.mjs`, `src/key-decoder.mjs`)** — Zero-dependency interactive selection, multi-select, secret masking, and raw keyboard decoding.
|
|
164
|
+
- [x] **Multi-OS CI Matrix Parity** — 100% green suite across Linux, macOS Darwin, and Windows CMD/PowerShell on Node 20, 22, and 24.
|
|
165
|
+
- [x] **Atomic Git Checkpoint & Rollback (`src/ops/checkpoint.mjs`, `agentctl rollback`)** — Pre-flight working tree snapshots and 1-command git restore.
|
|
166
|
+
- [x] **Documentation Sync Gate (`scripts/doc-sync-check.mjs`, `npm run jules:doc-sync`)** — Mechanical enforcement preventing documentation drift across manifests, roadmaps, and tests.
|
|
167
|
+
|
|
168
|
+
---
|
|
169
|
+
|
|
170
|
+
## 🎯 Target Milestones (v0.60.0 & v1.0.0)
|
|
171
|
+
|
|
172
|
+
### v0.60.0: Distributed File Leases & Preemptive DAG Scheduling
|
|
173
|
+
- [ ] **Atomic Filesystem Lease & Heartbeat Protocol (`src/engine.mjs`, `src/flaky-ledger.mjs`)** — Directory-mutex file leasing with heartbeat timestamps, stale-lock detection via PID liveness inspection, and tombstone rotation without third-party daemons or Redis.
|
|
174
|
+
- [ ] **Preemptive Task Cancellation & Interface Fingerprints (`src/dag-engine.mjs`)** — Automatically aborts and yields downstream swarm tasks when upstream exported symbol interfaces diverge from their cryptographic SHA-256 fingerprints.
|
|
175
|
+
- [ ] **POSIX/Win32 Process Group Guillotine (`src/engine.mjs`)** — Tree teardown via `process.kill(-pid, 'SIGKILL')` on POSIX and `taskkill /T /F /PID` on Windows to eliminate orphaned dev-servers and background watchers.
|
|
176
|
+
- [ ] **Unicode Trojan Source & Homoglyph Fencing (`src/security.mjs`)** — Deterministic token scanner using V8 Unicode Property Escapes (`\p{Script=...}`) and NFKC normalization to block invisible Bidi overrides (CVE-2021-42574) and mixed-script homoglyphs.
|
|
177
|
+
|
|
178
|
+
---
|
|
179
|
+
|
|
180
|
+
## 🏁 Target Milestone v1.0.0: The Production-Grade Autonomous Engineering Kernel
|
|
181
|
+
*Focus: Long-term API stability, cryptographic compliance, and enterprise deployment guarantees.*
|
|
182
|
+
|
|
183
|
+
- [ ] **Cryptographic Compliance & SOC2 Audit Exporter (`agentctl audit export`)**:
|
|
184
|
+
- Export tamper-evident, signed JSON-LD / SPDX receipts of all agent activities linked to the SHA-256 telemetry ledger.
|
|
185
|
+
- *Foundation shipped:* `agentctl evidence generate|verify|show` (`src/evidence.mjs`) already produces SHA-256 evidence manifests with test-tamper locking.
|
|
186
|
+
- [ ] **Zero-Dependency Core Freezing & Stability Guarantee**:
|
|
187
|
+
- 100% API stability for `index.mjs` SDK exports, CLI exit codes (0–8), and configuration schema (`.agent/config.yml`).
|
|
188
|
+
- [ ] **High-Concurrency Swarm Benchmarking (500+ Daily Sessions)**:
|
|
189
|
+
- Stress testing with 50+ concurrent worker slots across 100k+ file repositories with zero lock contention or memory leaks.
|
|
190
|
+
- [ ] **Comprehensive Multi-Language Enterprise Test Matrix**:
|
|
191
|
+
- Automated CI test fixtures for polyglot environments (Node, Python, Go, Rust, .NET, PHP, Java, Flutter).
|
|
192
|
+
- [ ] **OODA Attempt Diff Retention & Inspection (`.agent/state/ooda/*.patch`, `agentctl patch --attempt <n>`)**:
|
|
193
|
+
- Retains intermediate working tree diffs and failure traces across OODA repair turns so developers can inspect failed hypotheses when an agent exhausts its retry budget.
|
|
194
|
+
|
|
195
|
+
---
|
|
196
|
+
|
|
197
|
+
## 🔮 Post-1.0 Long-Term Horizon (v1.x+)
|
|
198
|
+
|
|
199
|
+
- **Proactive Telemetry Ingestion (Type III Situational Awareness)**: Ingest dev-server crash logs, APM traces, and Playwright test artifacts into auto-synthesized task envelopes for background diagnosis.
|
|
200
|
+
- **Cross-Repository Swarm Orchestration**: Orchestrate breaking API contract changes across multiple distinct git repositories with atomic synchronization.
|
|
201
|
+
- **Multimodal Visual Verification Loop**: Direct integration with headless browser video/screenshot streams for autonomous visual regression repairs.
|
|
202
|
+
- **Wasm-Powered Structural AST Invariant Engine**: In-memory WebAssembly tree-sitter bindings (zero npm dependencies) for deep multi-language semantic AST verification.
|
package/bin/agentctl.mjs
CHANGED
|
@@ -383,6 +383,7 @@ async function main() {
|
|
|
383
383
|
committed: { type: "boolean" },
|
|
384
384
|
fix: { type: "boolean" },
|
|
385
385
|
"allow-protected": { type: "boolean" },
|
|
386
|
+
"allow-unreadable-tests": { type: "boolean" },
|
|
386
387
|
// The tamper guard has always had an override — `allowTestModifications`
|
|
387
388
|
// — and it was reachable only from JavaScript. So a legitimate change
|
|
388
389
|
// of spec, which necessarily rewrites what a test expects, hit a
|
|
@@ -413,6 +414,7 @@ async function main() {
|
|
|
413
414
|
mode: selectedMode,
|
|
414
415
|
fix: values.fix,
|
|
415
416
|
allowProtected: values["allow-protected"],
|
|
417
|
+
allowUnreadableTests: values["allow-unreadable-tests"],
|
|
416
418
|
allowTestModifications: values["allow-test-modifications"],
|
|
417
419
|
allowTestChanges: values["allow-test-change"],
|
|
418
420
|
jsonReport: values["json-report"],
|
|
@@ -425,7 +427,17 @@ async function main() {
|
|
|
425
427
|
console.log(`-----------------------------------------------------`);
|
|
426
428
|
for (const p of res.phases) {
|
|
427
429
|
const status = p.ok ? "✅ PASS" : "❌ FAIL";
|
|
428
|
-
|
|
430
|
+
// Test tampering shares this phase, and its exit code, with the
|
|
431
|
+
// secret scanner. The codes are frozen, but reporting an assertion
|
|
432
|
+
// deletion under a bare "SECRETS" heading sent operators looking for
|
|
433
|
+
// a credential to rotate. Name what actually failed.
|
|
434
|
+
const tamperOnly =
|
|
435
|
+
p.phase === "secrets" &&
|
|
436
|
+
!p.ok &&
|
|
437
|
+
(p.findings || []).length > 0 &&
|
|
438
|
+
(p.findings || []).every((f) => f.type !== "SECRET_DETECTED" && !/SECRET/i.test(f.type || ""));
|
|
439
|
+
const label = tamperOnly ? "SECRETS (test integrity — no secret found)" : p.phase.toUpperCase();
|
|
440
|
+
console.log(` Phase [${label}] : ${status}`);
|
|
429
441
|
if (p.violations) {
|
|
430
442
|
p.violations.forEach((v) => console.log(` - Violation: ${v.file} (Rule: ${v.rule})`));
|
|
431
443
|
}
|
|
@@ -1601,6 +1613,13 @@ async function main() {
|
|
|
1601
1613
|
console.log(` Stack detected : ${res.stack}`);
|
|
1602
1614
|
console.log(` Oracle TestCmd : ${res.testCmd}`);
|
|
1603
1615
|
console.log(` Config written : ${res.configPath}`);
|
|
1616
|
+
} else if (res.reason === "NO_ORACLE_POSSIBLE") {
|
|
1617
|
+
// Declining is a usable answer. Writing a config whose very first run
|
|
1618
|
+
// asserts its own impossibility is not: it left the user following
|
|
1619
|
+
// the gate's own repair advice into a dead end.
|
|
1620
|
+
console.error(`❌ No oracle could be generated for this repository.`);
|
|
1621
|
+
console.error(` ${res.detail}`);
|
|
1622
|
+
process.exit(1);
|
|
1604
1623
|
} else {
|
|
1605
1624
|
console.log(`ℹ️ Repository already has a verification oracle (${res.testCmd}). Use --force to overwrite.`);
|
|
1606
1625
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "jules-orchestrator-kit",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.68.0",
|
|
4
4
|
"description": "Zero-dependency safety gatekeeper, test oracle generator, and multi-agent coordination protocol for autonomous coding agents — Google Jules, Claude Code, Codex and Gemini CLI.",
|
|
5
5
|
"repository": {
|
|
6
6
|
"type": "git",
|
|
@@ -32,7 +32,10 @@
|
|
|
32
32
|
"JULES_RULES_TEMPLATE.md",
|
|
33
33
|
".agent/rules/",
|
|
34
34
|
".agent/prompts/",
|
|
35
|
-
".agent/workflows/"
|
|
35
|
+
".agent/workflows/",
|
|
36
|
+
"CHANGELOG.md",
|
|
37
|
+
"ROADMAP_V1.md",
|
|
38
|
+
"AGENTS.md"
|
|
36
39
|
],
|
|
37
40
|
"scripts": {
|
|
38
41
|
"init": "node bin/init.js",
|
|
@@ -331,6 +331,15 @@ function compareSemver(a, b) {
|
|
|
331
331
|
|
|
332
332
|
// ---- CLI ----
|
|
333
333
|
const isMain = process.argv[1] && process.argv[1].endsWith("doc-sync-check.mjs");
|
|
334
|
+
// This gate compares the documentation against the test suite, which the
|
|
335
|
+
// published package does not contain. Run from an installed copy it reported
|
|
336
|
+
// six failures about files that were never meant to be there.
|
|
337
|
+
if (isMain && !existsSync(join(process.cwd(), "test"))) {
|
|
338
|
+
console.error("Documentation sync is a release gate for the repository, not a check on an installed package.");
|
|
339
|
+
console.error("It compares README/ROADMAP/CHANGELOG against the test suite, which is not shipped.");
|
|
340
|
+
console.error("Clone the repository to run it. To check an installed copy, run: npm run guard-reach");
|
|
341
|
+
process.exit(1);
|
|
342
|
+
}
|
|
334
343
|
if (isMain) {
|
|
335
344
|
const argv = process.argv.slice(2);
|
|
336
345
|
const flag = (name) => {
|
|
@@ -56,6 +56,7 @@ import {
|
|
|
56
56
|
SCOPE_CANARIES,
|
|
57
57
|
INNOCENT_EDITS,
|
|
58
58
|
UNREADABLE_DIALECTS,
|
|
59
|
+
CROSS_FILE_CASES,
|
|
59
60
|
} from "../src/guard-policy.mjs";
|
|
60
61
|
|
|
61
62
|
/**
|
|
@@ -185,6 +186,29 @@ const canaryResults = new Map();
|
|
|
185
186
|
);
|
|
186
187
|
}
|
|
187
188
|
|
|
189
|
+
{
|
|
190
|
+
const wrong = [];
|
|
191
|
+
for (const c of CROSS_FILE_CASES) {
|
|
192
|
+
const lines = [];
|
|
193
|
+
for (const f of c.files) {
|
|
194
|
+
lines.push(`--- a/${f.file}`, `+++ b/${f.file}`, "@@ -1,20 +1,20 @@", ` ${c.context}`);
|
|
195
|
+
for (const l of f.removed) lines.push(`-${l}`);
|
|
196
|
+
for (const l of f.added) lines.push(`+${l}`);
|
|
197
|
+
lines.push(` ${c.context}`);
|
|
198
|
+
}
|
|
199
|
+
const got = (checkTestTampering(lines.join("\n")).violations || []).map((v) => v.type).sort();
|
|
200
|
+
const want = [...c.expect].sort();
|
|
201
|
+
if (JSON.stringify(got) !== JSON.stringify(want)) {
|
|
202
|
+
wrong.push(`${c.id}: got ${JSON.stringify(got)}, policy says ${JSON.stringify(want)} (${c.why})`);
|
|
203
|
+
}
|
|
204
|
+
}
|
|
205
|
+
add(
|
|
206
|
+
"a move across files is judged as a move",
|
|
207
|
+
wrong.length === 0,
|
|
208
|
+
wrong.length ? wrong.join("; ") : `${CROSS_FILE_CASES.length} two-file diffs judged as declared`
|
|
209
|
+
);
|
|
210
|
+
}
|
|
211
|
+
|
|
188
212
|
// --- 4. Predicate mutants ---------------------------------------------------
|
|
189
213
|
{
|
|
190
214
|
const survivors = [];
|
package/scripts/run-tests.mjs
CHANGED
|
@@ -1,11 +1,28 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
import { readdirSync } from "node:fs";
|
|
2
|
+
import { readdirSync, existsSync } from "node:fs";
|
|
3
3
|
import { join } from "node:path";
|
|
4
4
|
import { spawn, spawnSync } from "node:child_process";
|
|
5
5
|
import { cpus } from "node:os";
|
|
6
6
|
|
|
7
7
|
const root = process.cwd();
|
|
8
8
|
const testDir = join(root, "test");
|
|
9
|
+
|
|
10
|
+
// The published package ships `scripts/` and not `test/`, so running
|
|
11
|
+
// `npm test` inside an installed copy crashed with a raw
|
|
12
|
+
// `ENOENT: no such file or directory, scandir '.../test'`. The suite is not
|
|
13
|
+
// missing; it was never part of the tarball. Say that, rather than leaving
|
|
14
|
+
// someone to work it out from a stack trace.
|
|
15
|
+
if (!existsSync(testDir)) {
|
|
16
|
+
console.error("No test/ directory here.");
|
|
17
|
+
console.error("");
|
|
18
|
+
console.error("If this is an installed copy of jules-orchestrator-kit, that is expected:");
|
|
19
|
+
console.error("the test suite is not part of the published package. Clone the repository");
|
|
20
|
+
console.error("to run it: git clone https://github.com/FullThrottle83/jules-orchestrator-kit");
|
|
21
|
+
console.error("");
|
|
22
|
+
console.error("To check an installed copy instead, run: npm run guard-reach");
|
|
23
|
+
process.exit(1);
|
|
24
|
+
}
|
|
25
|
+
|
|
9
26
|
const testFiles = readdirSync(testDir)
|
|
10
27
|
.filter((f) => f.endsWith(".test.mjs"))
|
|
11
28
|
.sort()
|
package/src/engine.mjs
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import { loadConfig, parseYaml, normalizeScope } from "./config.mjs";
|
|
2
2
|
import { isTestPath } from "./test-paths.mjs";
|
|
3
|
+
import { isPlaceholderTestScript } from "./stack-detector.mjs";
|
|
3
4
|
import { checkCollectionFloor } from "./ops/test-collection.mjs";
|
|
4
5
|
import { checkScope, scanDiff, scanBinaryPayloads, redactSecrets } from "./security.mjs";
|
|
5
6
|
import { changedFiles, diffBytes, diffText, binaryDiffEntries, symlinkChanges, showFromOrigin, runCmd } from "./git.mjs";
|
|
@@ -214,6 +215,7 @@ export async function gate(opts = {}) {
|
|
|
214
215
|
: parsed.verify?.min_tests !== undefined
|
|
215
216
|
? parsed.verify.min_tests
|
|
216
217
|
: config.verify.minTests,
|
|
218
|
+
tamperGuard: parsed.verify?.tamperGuard || parsed.verify?.tamper_guard || config.verify.tamperGuard,
|
|
217
219
|
scope: parsed.verify?.scope || config.verify.scope || "global",
|
|
218
220
|
timeoutMs: parsed.verify?.timeoutMs || parsed.verify?.timeout_ms || config.verify.timeoutMs,
|
|
219
221
|
};
|
|
@@ -328,6 +330,11 @@ export async function gate(opts = {}) {
|
|
|
328
330
|
root,
|
|
329
331
|
allowTestModifications: opts.allowTestModifications === true,
|
|
330
332
|
allowTestChanges: opts.allowTestChanges,
|
|
333
|
+
// Read from the base commit like every other trusted field: a repository
|
|
334
|
+
// opts out of the tamper guard deliberately and on the record, and an
|
|
335
|
+
// uncommitted edit must not be able to do it.
|
|
336
|
+
tamperGuard: trustedVerify.tamperGuard,
|
|
337
|
+
allowUnreadableTests: opts.allowUnreadableTests === true,
|
|
331
338
|
});
|
|
332
339
|
// A binary file reaches the scanner as one summary line, so its contents were
|
|
333
340
|
// never looked at — a NUL byte in front of a token was enough to hide it.
|
|
@@ -588,7 +595,22 @@ export async function gate(opts = {}) {
|
|
|
588
595
|
: { ok: true, count: null, runner: null, reason: null };
|
|
589
596
|
const emptySuite = !collectionFloor.ok;
|
|
590
597
|
|
|
591
|
-
|
|
598
|
+
// A command that cannot fail is not an oracle, wherever it is configured.
|
|
599
|
+
//
|
|
600
|
+
// `task create` already refuses one — "Unfalsifiable Task Rejected: Task
|
|
601
|
+
// must include a non-trivial verification test/build command" — while the
|
|
602
|
+
// gate approved against `verify.test: "true"` and printed an advisory. The
|
|
603
|
+
// same kit disagreed with itself about the same string. The collection
|
|
604
|
+
// floor cannot catch this: `true` states no count, and failing on "I could
|
|
605
|
+
// not tell" would break every unlisted runner. But a command that is
|
|
606
|
+
// *recognisably* a placeholder is not an unlisted runner.
|
|
607
|
+
const placeholderOracle =
|
|
608
|
+
verificationRequired &&
|
|
609
|
+
!missingOracle &&
|
|
610
|
+
Boolean(testResult?.command) &&
|
|
611
|
+
isPlaceholderTestScript(testResult.command);
|
|
612
|
+
|
|
613
|
+
const verifyOk = !failingCmd && !testTampered && !missingOracle && !emptySuite && !placeholderOracle;
|
|
592
614
|
|
|
593
615
|
// What actually broke. Without this the verify phase reported `ok: false` and
|
|
594
616
|
// nothing else — not the stage, not the exit code, not a line of output — so
|
|
@@ -635,7 +657,22 @@ export async function gate(opts = {}) {
|
|
|
635
657
|
"The gate approves a change because verification passed. Zero stages executed is not a pass.",
|
|
636
658
|
],
|
|
637
659
|
}
|
|
638
|
-
:
|
|
660
|
+
: placeholderOracle
|
|
661
|
+
? {
|
|
662
|
+
stageId: "placeholder-oracle",
|
|
663
|
+
command: testResult?.command || null,
|
|
664
|
+
exitCode: 0,
|
|
665
|
+
stdout: "",
|
|
666
|
+
stderr:
|
|
667
|
+
`The verification command is ${JSON.stringify(testResult.command)}, which cannot fail. ` +
|
|
668
|
+
"Nothing about this change was checked, so approving it would certify nothing. " +
|
|
669
|
+
"Point verify.test at a command that runs this repository's tests, or — if this repository " +
|
|
670
|
+
"intentionally uses only the scope and secret phases — set verify.required: false, which says so.",
|
|
671
|
+
diagnostics: [
|
|
672
|
+
"`agentctl task create` already refuses a task with this command; the gate now agrees with it.",
|
|
673
|
+
],
|
|
674
|
+
}
|
|
675
|
+
: emptySuite
|
|
639
676
|
? {
|
|
640
677
|
stageId: "empty-suite",
|
|
641
678
|
command: testResult?.command || null,
|
package/src/guard-policy.mjs
CHANGED
|
@@ -235,6 +235,94 @@ export const TAMPER_CANARIES = [
|
|
|
235
235
|
added: [" $this->markTestSkipped(\"later\");"],
|
|
236
236
|
expect: "TEST_SKIP_INJECTION",
|
|
237
237
|
},
|
|
238
|
+
// Dialects a second cold-start trial found silent. chai's `.to.equal(` is a
|
|
239
|
+
// dot chain where RSpec's is a space, so the RSpec branch never reached it;
|
|
240
|
+
// node-tap's assertions hang off whatever the sub-test callback named its
|
|
241
|
+
// argument, which is `ct` as often as `t`.
|
|
242
|
+
{
|
|
243
|
+
id: "weakening/chai",
|
|
244
|
+
file: "packages/core/test/signal.test.tsx",
|
|
245
|
+
context: "// signals",
|
|
246
|
+
removed: [" expect(s.value).to.equal(v);"],
|
|
247
|
+
added: [" expect(s.value).toBeDefined();"],
|
|
248
|
+
expect: "ASSERTION_WEAKENED",
|
|
249
|
+
},
|
|
250
|
+
{
|
|
251
|
+
id: "expectation/chai",
|
|
252
|
+
file: "test/signal.test.js",
|
|
253
|
+
context: "// signals",
|
|
254
|
+
removed: [" expect(s.value).to.equal(3);"],
|
|
255
|
+
added: [" expect(s.value).to.equal(-1);"],
|
|
256
|
+
expect: "ASSERTION_EXPECTATION_CHANGED",
|
|
257
|
+
},
|
|
258
|
+
{
|
|
259
|
+
id: "expectation/node-tap",
|
|
260
|
+
file: "tests/test-config.js",
|
|
261
|
+
context: "// config",
|
|
262
|
+
removed: [" ct.equal(process.env.BASIC, 'basic')"],
|
|
263
|
+
added: [" ct.equal(process.env.BASIC, 'BROKEN_OVERRIDE')"],
|
|
264
|
+
expect: "ASSERTION_EXPECTATION_CHANGED",
|
|
265
|
+
},
|
|
266
|
+
{
|
|
267
|
+
id: "skip-injection/node-tap-options",
|
|
268
|
+
file: "tests/test-populate.js",
|
|
269
|
+
context: "// populate",
|
|
270
|
+
removed: ["t.test('does not write over keys', ct => {"],
|
|
271
|
+
added: ["t.test('does not write over keys', { skip: true }, ct => {"],
|
|
272
|
+
expect: "TEST_SKIP_INJECTION",
|
|
273
|
+
},
|
|
274
|
+
// Skipping from inside the test body, which is how every one of these
|
|
275
|
+
// ecosystems actually does it. The decorator and annotation forms were
|
|
276
|
+
// covered; the standard library's own method was not, so a test could be
|
|
277
|
+
// silenced with the most ordinary call in the language.
|
|
278
|
+
{
|
|
279
|
+
id: "skip-injection/unittest-body",
|
|
280
|
+
file: "tests/test_app.py",
|
|
281
|
+
context: "# app",
|
|
282
|
+
removed: [],
|
|
283
|
+
added: [' self.skipTest("disabled")'],
|
|
284
|
+
expect: "TEST_SKIP_INJECTION",
|
|
285
|
+
},
|
|
286
|
+
{
|
|
287
|
+
id: "skip-injection/pytest-body",
|
|
288
|
+
file: "tests/test_app.py",
|
|
289
|
+
context: "# app",
|
|
290
|
+
removed: [],
|
|
291
|
+
added: [' pytest.skip("not now")'],
|
|
292
|
+
expect: "TEST_SKIP_INJECTION",
|
|
293
|
+
},
|
|
294
|
+
{
|
|
295
|
+
id: "skip-injection/raise-skiptest",
|
|
296
|
+
file: "tests/test_app.py",
|
|
297
|
+
context: "# app",
|
|
298
|
+
removed: [],
|
|
299
|
+
added: [' raise unittest.SkipTest("nope")'],
|
|
300
|
+
expect: "TEST_SKIP_INJECTION",
|
|
301
|
+
},
|
|
302
|
+
{
|
|
303
|
+
id: "skip-injection/mocha-body",
|
|
304
|
+
file: "test/app.test.js",
|
|
305
|
+
context: "// app",
|
|
306
|
+
removed: [],
|
|
307
|
+
added: [" this.skip();"],
|
|
308
|
+
expect: "TEST_SKIP_INJECTION",
|
|
309
|
+
},
|
|
310
|
+
{
|
|
311
|
+
id: "skip-injection/jest-todo",
|
|
312
|
+
file: "test/app.test.js",
|
|
313
|
+
context: "// app",
|
|
314
|
+
removed: [],
|
|
315
|
+
added: [' test.todo("adds two numbers");'],
|
|
316
|
+
expect: "TEST_SKIP_INJECTION",
|
|
317
|
+
},
|
|
318
|
+
{
|
|
319
|
+
id: "skip-injection/go-skipnow",
|
|
320
|
+
file: "internal/calc/calc_test.go",
|
|
321
|
+
context: "// calc",
|
|
322
|
+
removed: [],
|
|
323
|
+
added: ["\tt.SkipNow()"],
|
|
324
|
+
expect: "TEST_SKIP_INJECTION",
|
|
325
|
+
},
|
|
238
326
|
{
|
|
239
327
|
id: "skip-injection/xctest",
|
|
240
328
|
file: "Tests/CalcTests/CalcTests.swift",
|
|
@@ -391,6 +479,15 @@ export const INNOCENT_EDITS = [
|
|
|
391
479
|
added: [" 3, // the correct answer", " );"],
|
|
392
480
|
why: "line comments were never stripped — `copyCode(i)` left `pending` at the comment and the copy after the loop put it back",
|
|
393
481
|
},
|
|
482
|
+
{
|
|
483
|
+
id: "reformat-assertion-multiline/python",
|
|
484
|
+
file: "tests/test_encoding.py",
|
|
485
|
+
context: "# encoding",
|
|
486
|
+
lead: ["def test_int_bytes(value, expect):", " enc = int_to_bytes(value)"],
|
|
487
|
+
removed: [" assert enc == expect"],
|
|
488
|
+
added: [" assert (", " enc == expect", " )"],
|
|
489
|
+
why: "Black and Ruff do this; `assert (` alone on a line names no value, so a line-level count read a reformat as a weakening",
|
|
490
|
+
},
|
|
394
491
|
{
|
|
395
492
|
id: "rename-helper/node",
|
|
396
493
|
file: "test/calc.test.js",
|
|
@@ -551,3 +648,57 @@ export const MULTILINE_CANARIES = [
|
|
|
551
648
|
expect: "ASSERTION_EXPECTATION_CHANGED",
|
|
552
649
|
},
|
|
553
650
|
];
|
|
651
|
+
|
|
652
|
+
/**
|
|
653
|
+
* Diffs that touch two files at once, where the verdict depends on both.
|
|
654
|
+
*
|
|
655
|
+
* Assertion tracking is per file, so moving a test between files — ordinary
|
|
656
|
+
* refactoring, done constantly — read as deleting every assertion in it and
|
|
657
|
+
* was reported as CRITICAL tampering. The assertion still exists and still
|
|
658
|
+
* runs. What the removal check is for is verification that *disappeared*.
|
|
659
|
+
*
|
|
660
|
+
* `expect: []` means the whole diff must produce no finding.
|
|
661
|
+
*/
|
|
662
|
+
export const CROSS_FILE_CASES = [
|
|
663
|
+
{
|
|
664
|
+
id: "move-a-test-between-files",
|
|
665
|
+
context: "# ctx",
|
|
666
|
+
files: [
|
|
667
|
+
{
|
|
668
|
+
file: "test/test_app.py",
|
|
669
|
+
removed: [" def test_setattr(self):", " self.assertEqual(5, app.test)", " self.assertRaises(AttributeError, setattr, app, 't', 6)"],
|
|
670
|
+
added: [],
|
|
671
|
+
},
|
|
672
|
+
{
|
|
673
|
+
file: "test/test_html.py",
|
|
674
|
+
removed: [],
|
|
675
|
+
added: [" def test_setattr_moved(self):", " self.assertEqual(5, app.test)", " self.assertRaises(AttributeError, setattr, app, 't', 6)"],
|
|
676
|
+
},
|
|
677
|
+
],
|
|
678
|
+
expect: [],
|
|
679
|
+
why: "a move is not a removal",
|
|
680
|
+
},
|
|
681
|
+
{
|
|
682
|
+
id: "delete-a-test-outright",
|
|
683
|
+
context: "# ctx",
|
|
684
|
+
files: [
|
|
685
|
+
{
|
|
686
|
+
file: "test/test_app.py",
|
|
687
|
+
removed: [" def test_setattr(self):", " self.assertEqual(5, app.test)", " self.assertRaises(AttributeError, setattr, app, 't', 6)"],
|
|
688
|
+
added: [],
|
|
689
|
+
},
|
|
690
|
+
],
|
|
691
|
+
expect: ["ASSERTION_REMOVAL", "ASSERTION_REMOVAL"],
|
|
692
|
+
why: "nothing caught it on the way, so verification really did disappear",
|
|
693
|
+
},
|
|
694
|
+
{
|
|
695
|
+
id: "change-it-on-the-way-across",
|
|
696
|
+
context: "# ctx",
|
|
697
|
+
files: [
|
|
698
|
+
{ file: "test/test_app.py", removed: [" self.assertEqual(5, app.test)"], added: [] },
|
|
699
|
+
{ file: "test/test_html.py", removed: [], added: [" self.assertEqual(99, app.test)"] },
|
|
700
|
+
],
|
|
701
|
+
expect: ["ASSERTION_REMOVAL"],
|
|
702
|
+
why: "what arrived is a different claim; only an exact arrival is a move",
|
|
703
|
+
},
|
|
704
|
+
];
|