model-orchestrator 0.1.8 → 0.1.10

package/CHANGELOG.md

@@ -4,6 +4,38 @@ All notable changes to this project are documented here. The format follows [Kee
 
 ## [Unreleased]
 
+## [0.1.10] - 2026-09-07
+
+Closes the last two verification items on #11. Both had been described as needing vendor sign-ins or infrastructure that was not available. Both turned out to be doable with what was already here, and doing them found a real defect.
+
+### Fixed
+
+- **The generated weekly audit orphaned a temp file on every timeout.** A run killed by the unit's `TimeoutStartSec` dies on SIGKILL, so no trap and no cleanup line of ours can run, and its `reports/.audit-<stamp>-XXXXXX` file was left behind forever. The job now sweeps `.audit-*` older than a day at start. A day is far outside the unit's own 900s deadline, so a temp belonging to a run still in flight can never be swept. Found by actually starting the unit on Ubuntu; the previous text-only assertion could not see it.
+
+### Added
+
+- `test/fixtures/`: raw output captured from **real vendor CLI runs**, with `manifest.json` recording the vendor version, the exact flags, the exit code, and what each fixture proves. Every other judge test in this repository uses shapes written by hand. The capture earned itself immediately: real codex 0.153.4 emits an `item.completed` whose item is `type:"error"` (a skills-budget warning) *before* `turn.completed`, which no synthetic fixture contained, and real agy returns `"OK\n"` with a trailing newline. `test/fixtures.test.js` runs every judge against them.
+- `test/systemd/run-on-ubuntu.sh` and its README: starts the generated job as a **real systemd user unit** and proves that the timeout kills the whole cgroup (a detached grandchild does not survive it), that a failed rerun preserves the previous report, that a malformed gateway key exits 2 before anything is written, and that the new sweep removes an aged orphan while leaving a fresh one alone. 11/11 on Ubuntu 24.04.4 LTS, systemd 255. Not part of `npm test`, which has no systemd to run against.
+
+### Notes
+
+- `test/fixtures/README.md` states its own gaps rather than hiding them: qwen's success shape is still synthetic because its key was not present in the capture environment, and `claude` and `ollama` have no judge, so no fixture.
+
+## [0.1.9] - 2026-09-06
+
+### Added
+
+- `--version` and `-v` on the installer, printing the package version and exiting before anything else is validated, so it answers from a broken or half-configured directory.
+- A "Vendor version compatibility" section in the README, naming the exact vendor CLI version each lane was built against, and saying plainly that the installer checks a binary's presence and never its version. Closes the compatibility-statement item on #11.
+
+### Fixed
+
+- `-h` and `-v` are now parsed. Both were listed in the flag table but unreachable, because parsing required a `--` prefix and rejected every single-dash argument. Strictness is unchanged: `-x` is still `unexpected argument`, `--versionn` still `unknown flag`.
+
+### Changed
+
+- The live-canary question on #11 is answered by design rather than left open: the canary is `cli-run --doctor --run`, which runs on the user's machine against the user's own sign-ins. A maintainer-credential canary in CI would prove one machine works and bill per run, so it is documented as deliberately absent.
+
 ## [0.1.8] - 2026-09-06
 
 ### Added
@@ -129,6 +161,8 @@ First release.
 - Adversarial audit: two Codex rounds plus a two-engine review (Codex, Antigravity); findings and fixes in `docs/audit-brief.md`. After the review: subagents go to the project root (`--project`), snippet paths computed from `--dir`, lane sections rendered from the selection, a primary agent required, level 3 asks for API keys separately from CLIs, images and CLI installs pinned, an activation summary at the end of every install.
 
 [Unreleased]: https://github.com/aunysillyme/model-orchestrator/compare/v0.1.7...HEAD
+[0.1.10]: https://github.com/aunysillyme/model-orchestrator/compare/v0.1.9...v0.1.10
+[0.1.9]: https://github.com/aunysillyme/model-orchestrator/compare/v0.1.8...v0.1.9
 [0.1.8]: https://github.com/aunysillyme/model-orchestrator/compare/v0.1.7...v0.1.8
 [0.1.7]: https://github.com/aunysillyme/model-orchestrator/compare/v0.1.6...v0.1.7
 [0.1.6]: https://github.com/aunysillyme/model-orchestrator/compare/v0.1.5...v0.1.6

package/README.md

@@ -108,6 +108,28 @@ Most of what this package ships is text an agent is asked to follow. Be clear ab
 
 If you need a property in the third row to be enforced, that is a router, a policy engine or a sandbox, and this package does not claim to be one.
 
+### Vendor version compatibility
+
+**This package detects that a binary exists. It does not check its version, and a present binary is not a working lane.** `--doctor` reports presence, and with `--run` sends one lane a one-word canary; neither validates that the vendor's flags, output shape or auth still match what the generated files assume.
+
+The lane wiring and the output judges were written against these versions, which are the ones this release was exercised on:
+
+| Lane | Vendor | Version this release was built against |
+|---|---|---|
+| `claude` | Anthropic | 2.1.226 |
+| `codex` | OpenAI | codex-cli 0.153.4 |
+| `agy` | Google | 1.1.27 |
+| `grok` | xAI | 1.0.5 |
+| `hermes` | Nous Research | 0.20.0 |
+| `qwen` | Alibaba | 0.22.3 |
+| `ollama` | local | 0.33.3 |
+
+Recorded on 2026-09-06 from the maintainer's own installs, by running each CLI's own version flag. Nothing here is pinned: these CLIs ship breaking flag changes on their own schedules, so a newer version may work perfectly, or may change a flag the generated wiring passes. When a lane starts failing after a vendor upgrade, compare against this table first.
+
+**The live canary runs on your machine, with your credentials.** That is what `node bin/cli-run.mjs --doctor --run` is: it sends every enabled lane one tiny prompt through your own sign-ins and reports `canary ok` or `canary FAILED rc=` per lane. Run it after install, and again after any vendor upgrade.
+
+It deliberately does not run in this repository's CI. A canary is only meaningful against real credentials, and there are no credentials a maintainer could supply that would tell **you** anything about **your** lanes: your sign-ins, your quota, your vendor versions. A maintainer-credential canary in CI would prove one machine works and bill someone per run to do it. So CI runs the full suite against stub lanes on Ubuntu and macOS, Node 18/20/22, plus a packaged install into a clean consumer, and the live check ships to you instead.
+
 ## Principles the whole thing rests on
 
 1. **Route by capability tier, not model name.** Default down, escalate on evidence.

package/bin/cli.js

@@ -17,13 +17,18 @@ import { planFiles, writeFiles, resolveSelection, resolveTools, resolveApis, dir
 // turn a dry run into a real one.
 const SPEC = {
   level: 'value', ais: 'value', primary: 'value', dir: 'value', project: 'value', tools: 'value', apis: 'value',
-  yes: 'bool', force: 'bool', dry: 'bool', 'dry-run': 'bool', 'no-install': 'bool', 'no-tools': 'bool', 'no-apis': 'bool', 'upgrade-runtime': 'bool', 'update-docs': 'bool', list: 'bool', help: 'bool', h: 'bool'
+  yes: 'bool', force: 'bool', dry: 'bool', 'dry-run': 'bool', 'no-install': 'bool', 'no-tools': 'bool', 'no-apis': 'bool', 'upgrade-runtime': 'bool', 'update-docs': 'bool', list: 'bool', help: 'bool', h: 'bool', version: 'bool', v: 'bool'
 };
 export function parseArgs(argv) {
   const out = {};
   const errors = [];
   for (let i = 0; i < argv.length; i++) {
-    const a = argv[i];
+    let a = argv[i];
+    // The only single-dash forms, both bool: everything else stays strict, so a
+    // typo cannot be silently absorbed as a short flag. `-h`/`-v` were listed in
+    // SPEC before 0.1.9 but unreachable, because parsing required a `--` prefix.
+    if (a === '-h') a = '--help';
+    else if (a === '-v') a = '--version';
     if (!a.startsWith('--')) {
       errors.push(`unexpected argument: ${a}`);
       continue;
@@ -69,6 +74,14 @@ if (parsed.errors.length) {
 const flag = (name) => parsed.out[name] === true;
 const opt = (name) => (typeof parsed.out[name] === 'string' ? parsed.out[name] : null);
 
+// Answered before anything else is validated, so `--version` works from a
+// broken or half-configured directory: the one question a user asks when they
+// are about to file a bug must never depend on the rest of the run.
+if (flag('version') || flag('v')) {
+  console.log(GENERATOR_VERSION);
+  process.exit(0);
+}
+
 if (flag('help') || flag('h')) {
   console.log(`model-orchestrator: set up a model orchestrator for the AIs you actually have.
 
@@ -96,6 +109,8 @@ Flags
   --dry, --dry-run   print the plan, write nothing
   --no-install       never offer to run npm installs
   --list             print the catalog
+  --version, -v      print the version and exit
+  --help, -h         print this and exit
 `);
   process.exit(0);
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "model-orchestrator",
-  "version": "0.1.8",
+  "version": "0.1.10",
   "description": "A model orchestrator you can npm run: route every task to the cheapest AI that does it well, across one agent, many CLIs, or a whole virtual machine. Three levels, one installer that asks what you have access to.",
   "type": "module",
   "bin": {

package/templates/advanced/vm/jobs/weekly-audit.sh

@@ -18,6 +18,14 @@ RUNNER_SECS="${RUNNER_SECS:-600}"   # the model call; TimeoutStartSec in the uni
 cd "$INSTALL_DIR" || { echo "weekly-audit: $INSTALL_DIR missing" >&2; exit 2; }
 mkdir -p reports
 
+# Sweep temp files a previous run could not clean up. A run killed by the unit's
+# TimeoutStartSec dies on SIGKILL, so no trap and no cleanup line of ours can
+# run, and its `.audit-<stamp>-XXXXXX` file is orphaned in reports/ forever.
+# Measured on Ubuntu 24.04 / systemd 255, 2026-09-07: one orphan per timeout.
+# Anything older than a day cannot belong to a live run (the unit's own deadline
+# is 900s), so removing it is safe even if another run is in flight.
+find reports -maxdepth 1 -name '.audit-*' -type f -mtime +0 -delete 2>/dev/null || true
+
 # bounded SECS cmd...  : run cmd, and after SECS kill it AND every descendant
 # (a probe that forks, or a stub that ignores its own flags, must not hold a
 # pipe open). Process groups do not help here: bash disables job control inside