@azure-id/orc 1.4.2 → 1.5.0

package/CHANGELOG.md

@@ -10,6 +10,117 @@ Format: `### v<version> — <title> _(<date>)_`.
 
 ---
 
+### v1.5.0 - the lane that runs the test _(2026-09-07)_
+
+**Still on the unscoped `orc` package?** Do this once first - your `orc upgrade`
+is the pre-v0.56.0 one and cannot install itself. Full detail in the CAUTION at
+the top of this file.
+
+- **Step 1 - release the command from the old package:** `npm uninstall -g orc`
+- **Step 2 - install the current package:** `npm i -g @azure-id/orc`
+- **Step 3 - re-apply it to your project:** `orc update`
+
+**Do not use `npm i -g -f`.** Full detail in v0.56.0 below.
+
+Every other testing surface in ORC **writes** tests. `/orc-test` **runs** them.
+
+It points at a running API - yours, on your laptop, or one you were given
+permission to touch - sends real requests, writes down what came back, and
+stops. The split is the whole design:
+
+> **The CLI executes and measures. The model designs and interprets.**
+> A model never sends a request. The CLI never decides what a response means.
+
+That split is what makes the report worth reading. A number in it was measured
+by a program. A sentence in it was written by a model that only ever saw
+evidence on disk.
+
+**Three rules hold it together**, each a registered contract:
+
+- **`a lane that reports a result it did not observe`** - three verdicts, and
+  `unknown` is the honest one. A case nobody ran is never a pass.
+- **`a lane that sends traffic to a target nobody authorized`** - the target is
+  frozen once, a remote one **requires** a written statement of who said yes,
+  and every request is fenced to one origin.
+- **`a lane that fixes the system under test`** - it reports and hands back. Not
+  to fix a boot failure, not to add a `data-testid`, not to relax a check.
+
+**The free pass runs first, and always.** `orc test surface` reads your
+repository and the target's own spec document for zero model tokens, and the
+**code-vs-live diff** falls out of it: a route that answers at the target and is
+in **no file in your repository** is undocumented surface nobody reviews and
+nobody patches. It is derived from the two spec documents and from nothing else
+- no route is ever called to produce it. With no spec at the target the diff
+reads **NOT MEASURED, with the reason**, because an empty list there would say
+"no shadow APIs", which is a claim nobody made.
+
+**The case matrix derives itself.** The tiers come out of the schema for free.
+What costs anything is a **gap** - what a schema cannot know, like what a valid
+email looks like in your business, or which order two calls have to happen in.
+That is the only part a model is paid for.
+
+**The security tier is a closed set** - the OWASP API Top 10 (2023), ten rows,
+never extended on the fly, and **all ten render in every state**. A category ORC
+could not measure says `UNCHECKABLE`, **keeps its slot**, carries its own
+reason, never becomes a pass, and never changes the exit code. Reporting an
+unmeasured category as clean is the single most damaging thing a security report
+can do, because somebody then ships on it. It **detects and never exploits**:
+every probe shows the condition and stops there.
+
+**When your app is broken, it stops.** `orc test env` finds the start command,
+sees the process is alive, reads the exact line in the log - and **will not open
+that file**. It names the environment values that are absent and writes a value
+for none of them, because a placeholder in a real environment is a
+credential-shaped lie and only you know the right one.
+
+**The runner is the only place in ORC that sends a request to a system it did
+not start**, so everything in it exists to make sure ORC's own run is never the
+incident. A red happy path **stops the ladder** - a 500 on every request will
+"prove" a dozen vulnerabilities that are one bug. The pace is the target's, and
+the CLI applies it. **A 429 is a result**: it means rate limiting works, so ORC
+records it and backs off rather than pushing through, which is how a staging
+scan becomes a lockout. Failed logins are capped per identity. **Redaction is
+structural** - every credential is replaced before the bytes reach disk, not in
+a review step afterwards.
+
+**A finding has to point at something.** One that cites evidence which does not
+resolve on disk is **dropped by name and counted**. Severity is ORC's; if the
+interpreting model disagrees, its word is kept beside ORC's and printed, so a
+disagreement is visible instead of invisible. **A flake is recorded, never
+retried away** - the instability is the finding.
+
+**The front end drives Playwright as a script**, not as an agent loop: a step an
+LLM took is not a step you can re-run. ORC **names** the install command and
+never runs it - it does not install into your project.
+
+**`orc ui` ▸ Test** renders all of it and works out none of it. Five tabs, a
+button for every free action, and **`orc test run` as a command you copy** -
+never a button. It costs no model tokens, so the usual rule would make it one;
+it sends real traffic, and a page that can start a scan against a live host is a
+page that can start one by accident.
+
+**The run folder is never staged.** It holds real response bodies from a real
+system - the most sensitive thing ORC writes to disk. `orc doctor` tells you if
+it is not git-ignored, and it will not edit `.gitignore` for you.
+
+Five config keys: `test_gate` (warn only - the payoff is knowing, not gating),
+`test_security_tier`, `test_max_rps`, `test_case_budget`, `test_ui_driver`. Five
+more were **refused**, with the reasons written down so nobody proposes them
+again - among them `test_retries`, because a flake is recorded rather than
+retried away, and `test_auto_fix`, which is this lane's premise inverted.
+
+**And one bug that was not in this lane.** Writing the tests isolated an
+intermittent failure that turned out to be real: `orc ui` compressed its API
+answers but not its **static files**, guarded by a comment reading *"No asset is
+over 64 KiB today"* that had been false for two releases. `js/panels/extra.js`
+is 138 KB. Over a Windows loopback socket a response that size loses its tail
+roughly one time in six - which in a browser is a panel script that stops in the
+middle of a function, with nothing in any log to say so. Both paths now share
+one encoder. **A stale comment can guard a live bug for longer than the bug
+would have survived alone.**
+
+---
+
 ### v1.4.2 - the panel that stops reloading, and the fields you could not read back _(2026-09-05)_
 
 **Still on the unscoped `orc` package?** Do this once first - your `orc upgrade`

package/README.md

@@ -14,7 +14,7 @@
 ![Dependencies](https://img.shields.io/badge/dependencies-zero-lightgrey.svg?style=for-the-badge)
 ![GitHub stars](https://img.shields.io/github/stars/azure-id/orc?style=for-the-badge&color=yellow)
 
-**Latest: v1.4.2** · updated 2026-09-05 · [full changelog](CHANGELOG.md)
+**Latest: v1.5.0** · updated 2026-09-07 · [full changelog](CHANGELOG.md)
 
 **On npm: [`@azure-id/orc`](https://www.npmjs.com/package/@azure-id/orc)** — `npm i -g @azure-id/orc`
 
@@ -287,6 +287,7 @@ ORC have terminal hook to see: Context Window %, 5 Hour usage %, Weekly usage %
 | **`/orc-challenge`** | Grades a **finished** artifact — a TSD, a PRD, an ADR, an API contract, a README, a module — against a goal **you** state, then **stops and makes you fix it in a different session**. ORC judges, you fix, ORC re-judges: **it never fixes what it judged**, because a session that just wrote the fix would grade its own homework. **And it never guesses what "good" means here.** | [see it](mock-run/orc-challenge.md) |
 | **`/orc-pact`** | The promises your project makes, and which are in doubt right now. Four states, all **computed on read**: HOLDING · DRIFTED · **UNCHECKABLE** (the honest state — never a failure) · BROKEN. It never invents a promise and never retires one for you. | [see it](mock-run/orc-pact.md) |
 | **`/orc-boundary`** | What ORC should **not** try here, and exactly what would change that. EXECUTE · ESCALATE · REFUSE, per area. **A REFUSE always names what would make it a yes.** It gates ORC's own dispatch, never your instructions. | [see it](mock-run/orc-boundary.md) |
+| **`/orc-test`** | **Runs** the test against a running system - yours, or one you were given permission to touch. Sends the requests, writes down what came back, and stops. **The CLI executes and measures; the model designs and interprets.** Three verdicts, and **`unknown` is the honest one**. The OWASP API Top 10 (2023) as a **closed set** where a category it could not measure keeps its slot and never becomes a pass. It never edits the system it is testing. | [see it](mock-run/orc-test.md) |
 | **`/orc-verify`** | Verifies only your git-modified changes: build, tests, diff sanity, findings on a P0–P3 ladder. Read-only. | [see it](templates/skills/orc-verify/examples/verify-mock.md) |
 | **`/orc-aftermath`** | Did what we shipped hold up? Graded from the repository's own future: churn, reverts, deleted tests, broken promises. **Churn is a signal, never a verdict**, and it never names a person. | [see it](mock-run/orc-aftermath.md) |
 | **`/orc-budget`** | What a run costs, in the unit you are billed in. A **token vector** — fresh input, cache write, cache read, output, never blended — shown four ways: tokens, dollars, percent of your 5-hour window, and context risk. Needs a plan, not a sentence. | [see it](mock-run/orc-budget.md) |
@@ -575,7 +576,7 @@ a current audit: [EVAL-REPORT.md](EVAL-REPORT.md).
 **Full history: [CHANGELOG.md](CHANGELOG.md)** — or `orc changelog`, which prints
 only what is newer than the version you have.
 
-### v1.4.2 - the panel that stops reloading, and the fields you could not read back _(2026-09-05)_
+### v1.5.0 - the lane that runs the test _(2026-09-07)_
 
 **Still on the unscoped `orc` package?** Do this once first - your `orc upgrade`
 is the pre-v0.56.0 one and cannot install itself. Full detail in the CAUTION at
@@ -587,50 +588,49 @@ the top of this file.
 
 **Do not use `npm i -g -f`.** Full detail in v0.56.0 below.
 
-Four more defects in `orc ui` ▸ **CLI Hook Interface**. Three were the same
-defect wearing three faces.
-
-**Staging a change reloaded the whole panel** - every staged op re-entered the
-router, so four endpoints were refetched and everything was rebuilt from a
-skeleton. That made a staged change look like an applied one, and it left the
-open editor showing values from before the change you had just made: you picked
-a shape and nothing moved. Nothing is fetched now until something actually moves
-the disk; a staged op repaints from cache with **zero requests**, keeps your
-scroll position, and the open editor repaints with it - caret and all.
-
-**And the fields could not be read back.** `orc statusline show --json` emitted
-twelve of the twenty-four fields a part can carry, so `case`, `prefix`,
-`min_cols`, `precision` and seven more were written to disk correctly and then
-were invisible to the panel that wrote them. Every field ships now, plus
-`authored`, so a control can say **which values are yours** rather than
-inherited. `--json is not a summary`, found again.
-
-**Six parts per line, not five.** **The dropdown nobody could read** turned out
-to be a panel with no colours at all: `hookui.css` was written against four CSS tokens that do not
-exist, so every one of those declarations was silently dropped.
-
-**The preview is drawn as a terminal now** - a window of the stated width with a
-column ruler - and `orc statusline preview` takes `--theme` and `--glyphs` as a
-**render-only** override, so a new drawer shows the same bar under all four
-colour sets and all seven symbol sets. Picking a colour set from its name is a
-guess; picking it from its picture is a decision.
-
-**And the sixth part still reset the bar to the default**, because the cap was in
-three places: the hook re-checks the compiled file with a cheap shape guard of
-its own, and that guard held its own `5` — and counted spacers, which the
-validator never has. The cap now travels on the lock file and the compiler marks
-each structural item, so one cap and one counting rule serve all three readers.
-
-**And a name you typed sometimes went nowhere.** Seven of the thirty-five shapes
-draw a name; the other twenty-eight ignore it, which is the design — a bar with a
-word in front of it is a different shape. The Name box was offered on all
-thirty-five anyway. Each renderer now publishes the fields it **uses**, and the
-editor greys out what this shape ignores, keeping the slot and naming the shapes
-that do.
-
-**And it still blinked with the network tab empty:** `hkPaint` replaces every
-child, and the stylesheet faded each one in on a stagger. Entrance animations are
-off from the second paint onward.
+Every other testing surface in ORC **writes** tests. `/orc-test` **runs** them.
+
+It points at a running API - yours, on your laptop, or one you were given
+permission to touch - sends real requests, writes down what came back, and
+stops.
+
+> **The CLI executes and measures. The model designs and interprets.**
+> A model never sends a request. The CLI never decides what a response means.
+
+That split is what makes the report worth reading: a number in it was measured
+by a program, and a sentence in it was written by a model that only ever saw
+evidence on disk.
+
+**The free pass runs first.** `orc test surface` reads your repository and the
+target's own spec document for zero model tokens, and the **code-vs-live diff**
+falls out of it - a route that answers at the target and is in **no file in your
+repository** is surface nobody reviews and nobody patches. With no spec at the
+target it reads **NOT MEASURED, with the reason**, because an empty list there
+would say "no shadow APIs", which is a claim nobody made.
+
+**The security tier is a closed set** - the OWASP API Top 10 (2023), and all ten
+rows render in every state. A category ORC could not measure says `UNCHECKABLE`,
+keeps its slot, and **never becomes a pass**. It detects and never exploits.
+
+**When your app is broken it stops.** It finds the start command, sees the
+process is alive, reads the exact line in the log - and will not open that file.
+
+**A red happy path stops the ladder**, because a 500 on every request will
+"prove" a dozen vulnerabilities that are one bug. A **429 is a result** - it
+means rate limiting works, so ORC backs off rather than pushing through. Every
+credential is replaced **before the bytes reach disk**. A finding that cannot
+point at evidence on disk is **dropped by name**. A flake is **recorded, never
+retried away**.
+
+**`orc ui` ▸ Test** renders all of it and works out none of it - and `orc test
+run` is a command you **copy**, never a button: a page that can start a scan
+against a live host is a page that can start one by accident.
+
+**And one bug that was not in this lane.** Writing the tests isolated a real
+one: `orc ui` compressed its API answers but not its static files, guarded by a
+comment reading *"No asset is over 64 KiB today"* that had been false for two
+releases. Over a Windows loopback socket a 138 KB panel script loses its tail
+roughly one time in six. Both paths share one encoder now.
 
 Full detail in [CHANGELOG.md](CHANGELOG.md).