opencode-harness-kit 0.4.2 → 0.4.4

package/CONTRIBUTING.md

@@ -0,0 +1,242 @@
+# Contributing to opencode-harness-kit
+
+Thanks for looking. This is a small project and the bar for a change is that it
+is either verifiable or it does not land.
+
+## Before you start
+
+`npm run check` must pass on an untouched checkout. If it does not, that is a
+bug worth its own issue — do not build on top of a red baseline.
+
+`npm run check` is the **offline** gate: build, typecheck, tests. It needs
+nothing but the repository, so it is green on a machine that has never seen an
+OpenCode login.
+
+`npm run check:install` is the other half, and it needs a machine that HAS run
+OpenCode: it measures the install database (`sqlite3` and
+`~/.local/share/opencode/opencode.db`), reads the installed config, and
+verifies it. It cannot run on a fresh checkout, and it is not part of
+`npm run check` for that reason. CI runs the offline half; the live half is
+yours to run before a release.
+
+## What a change has to come with
+
+The test suite is the contract, so a change without a test is a change nobody
+can trust:
+
+- **Model routing, effort tiers, pruning, retention.** Add an assertion to
+  `test/`. These tests run offline and must stay offline: no network, no model
+  calls, no API keys. `npm test` has to pass on a machine that has never seen
+  an OpenCode login.
+- **A new command or flag.** `oc-harness <flag>` that the docs do not mention
+  will be found by nobody, and `oc-harness help` listing a command that does not
+  exist is worse.
+- **A changed number.** If your change moves a number a profile claims —
+  effort, context budget, token budget — the README has to change with it, in
+  the same commit. A stale number in the README is a bug, not a nit.
+- **A change to the grader.** The grader decides what the A/B means, and a
+  number produced by a grader nobody has read is a number nobody should
+  quote. `test/verify.test.js` holds the cases that actually happened.
+- **Anything that writes to disk.** It needs a `--dryrun` path, and it must keep
+  a backup. `~/.config/opencode/` is a live config for anyone who runs it,
+  including the people reviewing your PR.
+
+## Running the checks
+
+```bash
+npm run check        # build + typecheck + tests, offline
+npm run check:install  # the above, plus the live install measurement and verify
+npm test             # the offline assertions, fast
+npm run typecheck    # the library, then the scripts
+npm run typecheck:scripts  # only bin/ and eval/
+```
+
+`bin/` and `eval/` are type-checked too, with a looser config than the
+library: they are plain JavaScript with no JSDoc on their parameters, and
+annotating a few hundred of them would be churn rather than work. The loose
+config still catches a wrong type and a missing property on a declared
+shape; it does not catch a property that does not exist on an INFERRED
+object literal - TypeScript does not check those in a `.js` file - so an
+object you build and then add to wants a `@type` on it. The comment in
+`tsconfig.scripts.json` says which is which.
+
+`npm run check:install` measures an existing OpenCode install and verifies
+the config in `~/.config/opencode/`. It does not install anything itself, and
+it reads your live config. If you are not comfortable with that over your own
+config, run the layers individually and say so in the PR.
+
+`eval/ab.mjs` spawns real agents and costs real tokens. It is deliberately
+not part of any check, and a PR does not need it. `npm run measure` does not
+spawn an agent either — it queries the install database read-only.
+
+## House rules
+
+- Node 22 or newer. The CLI uses `node:` prefixed builtins, so an older Node
+  fails at import time rather than with a useful message.
+- No new runtime dependencies without a reason that survives review. A CLI that
+  installs is one a lot of people will not install twice.
+- `dist/` is committed because the release tarball needs it. It is generated by
+  `npm run build`; never hand-edit it, and never commit it without rebuilding.
+- Match the surrounding comment style: comments explain **why**, especially
+  where the obvious version would be a trap.
+
+## Pull requests
+
+Keep the diff to one concern. Say what you changed, what you measured, and what
+you did not check. If you fixed a bug, the PR should fail before it and pass
+after — say which test does that.
+
+The version is set in `package.json` by `npm version`, and the tag is what
+triggers the publish. Keep the two in step: the job publishes the version in
+`package.json`, so a tag that says one version and a file that says another
+is refused by the registry with `EPUBLISHCONFLICT`.
+
+## Cutting a release
+
+```bash
+# 1. the version and the changelog entry
+npm version minor            # or major, or patch
+$EDITOR CHANGELOG.md        # the entry for the new version
+
+# 2. the tag, which is what triggers the publish
+git commit -am "release: <version>"
+git tag -a v<version> -m "<what the release is>"
+git push origin main v<version>
+```
+
+The tag triggers the workflow, the tests run, and the `publish` job publishes
+the tarball. `npm version` writes the version into `package.json`, and the
+publish job publishes the version in `package.json` — so the tag and
+`package.json` have to agree, or the registry refuses the publish with
+`EPUBLISHCONFLICT`. That is the hazard worth guarding: not the bump, but a
+tag that says one version and a file that says another.
+
+### How the publish authenticates
+
+Two ways, and the job picks whichever is available:
+
+**Trusted publishing (OIDC)** — the default, and what the repository is set up
+for. There is no npm secret in the repository at all: npm exchanges the
+runner's OIDC identity for a short-lived publish token scoped to the workflow.
+
+**A token** — the fallback, for when OIDC is not set up. Set the `NPM_TOKEN`
+repository secret to a granular token scoped to `opencode-harness-kit` with
+**Read and write** on the package. If the secret is present the job uses it;
+if it is absent the job uses OIDC.
+
+```bash
+gh secret set NPM_TOKEN --repo pham-canh/space-harness
+```
+
+> **`gh secret set` takes the secret NAME as its first argument**, and the
+> value is pasted at the prompt. `gh secret set <the-token-value>` creates a
+> secret whose *name* is the token - and the name of a secret is visible to
+> everyone who can read the repository's Actions secrets list. If a secret
+> ever shows up named like `npm_...`, delete it and set it again properly.
+
+> **A token that can read the package is not necessarily a token that can
+> publish it.** With two-factor authentication on the account, a granular
+> token has to be created with **Bypass two-factor authentication** ticked or
+> the publish is refused with `EOTP - This operation requires a one-time
+> password`, and the status is a plain **401**. `npm whoami` passes, because
+> reading needs no second factor. A 401 on the publish with a working
+> `npm whoami` means the token needs the box ticked, not more permissions.
+
+A token needs write access *on the package*, not on the account. npm answers
+**404** for a package the token cannot touch, which reads like a missing
+package rather than a missing permission.
+
+### Configuring the trusted publisher
+
+npm's web form has four fields and **npm does not validate what you type into
+it** — a typo is discovered at publish time, and npm reports it as
+
+```
+OIDC token exchange error - package not found
+```
+
+which reads like a missing package. The package is there; the configuration
+is not matching, and npm does not say which field.
+
+npm ships `npm trust` for exactly this, and it can also read what is stored.
+So the configuration is set from the command line, where it is compared
+against the identity the workflow will actually present:
+
+```bash
+npm install -g npm@latest      # npm trust needs 11.15.0 or later
+npm login                       # 2FA has to be enabled on the account
+oc-harness trust                # check, and replace it if it is wrong
+```
+
+npm completes a two-factor challenge in a **browser**, and the command opens
+the browser for it on the platform it is running on — `open` on macOS,
+`xdg-open` on Linux, `rundll32` on Windows. Answer there however the account
+is set up — an authenticator app, a security key, or a code sent to the
+email. The URL is printed as well, for a machine with no browser to open, and
+the command says which of the two happened rather than claiming the page is
+open when it is not.
+
+A code can also be given on the command line, which skips the browser:
+
+```bash
+npx -y opencode-harness-kit@latest trust --otp=123456
+```
+
+The wait for a human is `OC_TRUST_WAIT_SECONDS` (600 by default).
+
+`oc-harness trust` reads what npm has stored, compares it against the
+workflow's own identity, and replaces it when it differs. npm allows exactly
+one configuration per package and cannot edit it, so replacing means revoking
+first — the command does that.
+
+It works from any directory, which matters: a release tool that only runs
+from one place is a release tool that people run from the wrong one, and the
+failure is `Cannot read package.json`, which says nothing about what the
+command was for.
+
+To set it without the script, the command is:
+
+```bash
+npm trust github opencode-harness-kit \
+  --file ci.yml \
+  --repo pham-canh/space-harness \
+  --allow-publish --allow-stage-publish -y
+```
+
+The configuration it writes is:
+
+| field | value |
+|---|---|
+| Organization or user | `pham-canh` |
+| Repository | `space-harness` |
+| Workflow filename | `ci.yml` (the filename only, not a path) |
+| Environment name | **empty** |
+| Allowed actions | `npm publish` and `npm stage publish` |
+
+**The environment name is the field that catches people.** GitHub only puts an
+`environment` claim in the OIDC token when the job declares one. This
+workflow declares none, so the claim is absent — and a configuration that
+names an environment can never match a token that does not carry one.
+
+The workflow is the one in `.github/workflows/ci.yml`, and the claims it
+presents are printed by the publish job before it tries:
+
+```
+repository    pham-canh/space-harness
+workflow file ci.yml
+environment   (absent) - the trusted publisher must name NO environment
+owner         pham-canh
+```
+
+A newly created trusted publisher has to complete its first successful
+publish **within 2 days**, or it expires and has to be created again.
+
+> **Both `npm trust` and `npm publish` need a one-time password**, and the
+> web URL they print is masked in the log — so run them in a terminal, where
+> npm prompts for the code. The token path below needs none of that.
+
+The GitHub release is written by hand from the CHANGELOG entry:
+
+```bash
+gh release create v<version> --title "<version> - <what it is>" --notes-file <notes>
+```

package/GLOBAL.md

@@ -12,11 +12,31 @@ worse, not better.
 ## Verify, then claim
 
 A task is finished when its verification command passes, not when the code
-looks right. Run the project's own test, lint and typecheck command. Paste the
-real result, including failures. If you cannot run it, say so plainly.
+looks right. Paste the real result, including failures. If you cannot run it,
+say so plainly.
 
 Never invent a test result, a benchmark number, a deploy URL, or a file path.
 
+**Run the check that covers what you changed, not every check the project
+has.** This is the rule that decides whether the harness is worth having, and
+it was measured:
+
+- An absolute rule - "run the project's test, lint and typecheck command" -
+  made an agent run the whole suite, and install the dependencies, to rename one
+  constant. Across 60 measured runs it was **11.4 bash calls per task against
+  0.2 without it**, and it made the answers *worse*, because every extra
+  command is another chance to wander off.
+- The narrow rule is: if the task names a verify command, run **that** one. If
+  it does not, run the smallest command that covers the code you touched, and
+  say which one you ran. If you cannot tell which covers it, run the whole
+  suite once - once, not three times.
+- Do not re-run a check that already passed. One run is the result.
+
+**Do not install anything.** Not `npm install`, not `npm ci`, not a package
+the task did not ask for. Installing runs code you did not read, and it
+changes the tree you were asked to change. If the task needs a dependency
+that is not there, say so.
+
 ## Decide, don't ask
 
 This is the rule that most changes how a session goes, so it is stated as a
@@ -87,11 +107,20 @@ to avoid this.
 2. `read` with an `offset`/`limit` for the range you need.
 3. Read a file whole only when you are about to edit it.
 
+**When the question already names the file, read it.** The rule above is for
+the case where you do not know where something is. Searching for a file you
+were just told the name of is two extra round trips for the same answer, and
+across 60 measured runs the absolute form cost 3.7 `grep` and 4.8 `read` calls
+a task against 1.5 and 1.2 for the conditional one.
+
+Read the whole file when you are going to edit it. Read a range when you are
+going to cite a line.
+
 ## Never
 
-- Force-push, rewrite published history, delete a branch or tag, or publish to
-  a registry (`npm publish`, `gh release`). Push without being asked; ask before
-  any of these, because they are the ones with no undo.
+- Force-push, rewrite published history, delete a branch or tag, open a pull
+  request, or publish to a registry (`npm publish`, `gh release`). Push without
+  being asked; ask before any of these, because they are the ones with no undo.
 - Commit a secret. If you find one, report it; do not move it.
 - Read or write `.env`, `.npmrc`, `.netrc`, `.git-credentials`. The guard
   blocks these and will keep blocking them.

package/MIGRATION.md

@@ -1,3 +1,154 @@
+# Migrating
+
+Two upgrades change what a user has to do. The most recent one first.
+
+- [0.3.x → 0.4.x](#migrating-from-03x-to-04x) — the kit starts enforcing
+  things it used to only suggest, and the config says which kit wrote it.
+- [0.2.x → 0.3.0](#migrating-from-02x-to-030) — the single-model profile
+  became a fleet.
+
+---
+
+# Migrating from 0.3.x to 0.4.x
+
+**Nothing you have breaks.** No profile id changed, no file moved, and no
+config field was removed. What changed is that the kit started enforcing
+things it used to only suggest, and it started saying which version wrote
+your config.
+
+## 1. The guard now blocks what it used to allow
+
+The denylist was a list of patterns the rules asked the agent to respect.
+It is now checked in the code, before the command runs:
+
+| Command | 0.3.x | 0.4.x |
+|---|---|---|
+| `npm publish` | allowed, with a warning in the rules | **refused** |
+| `npm install`, `npm ci` | allowed | **asks first** |
+| `git push` | allowed | **asks first** |
+| `gh pr create`, `gh pr merge` | allowed | **refused** |
+
+If one of those was load-bearing in your workflow, the honest answer is
+that it was never going to be the kit's call. Nothing is refused silently:
+the reason is printed, and the command is not run.
+
+## 2. Your config records the kit that wrote it
+
+```jsonc
+"harness": {
+  "profile": "fleet-safe",
+  "kit": "0.4.4"
+}
+```
+
+Without it, a config from three releases ago and one from today are the same
+file to `verify`, and the only way to tell them apart is to remember. If
+`kit` is missing, your config was written before 0.4.0 and is still valid —
+re-run `oc-harness use <profile>` to bring it up to date.
+
+## 3. The pre-commit hook runs your project's tests, not the kit's
+
+This is the change most likely to surprise you. The hook used to run the
+kit's own test suite against the kit's own repository. In a project of
+yours, that suite passed or failed for reasons that had nothing to do with
+you — and it blocked every commit in the process.
+
+The hook is now installed only where it belongs:
+
+- in the kit's own repository, and
+- in a project of yours, if you ask for it with `oc-harness install --hook`.
+
+Where it is installed, it runs **your** project's test script, and only that.
+
+```bash
+oc-harness install --hook
+```
+
+## 4. The kit works in a directory that is not a git repository
+
+The kit used to find the project root through the git worktree. In a
+directory that is not a repository, that lookup returned the filesystem root
+- so the kit found no project, enabled nothing, and reported itself healthy
+while doing nothing at all.
+
+The project root is now the directory you are in. If you want it to find
+the repository above you, run the kit from inside it.
+
+## 5. `OPENCODE_HARNESS_PROFILE` names a profile, not a path
+
+The variable was checked as if it were a file path, so it never resolved
+and the override never took effect. It names a profile id:
+
+```bash
+export OPENCODE_HARNESS_PROFILE=fleet-safe
+```
+
+## 6. The guard reads the whole command line
+
+The denylist used to be matched against the raw command, so the commands it
+names could be reached through a wrapper. They cannot any more:
+
+```
+cd /tmp && npm publish     blocked
+sudo npm publish           blocked
+sh -c "npm publish"        blocked
+sudo bash -c "npm publish" blocked
+```
+
+The rules the kit already had - matched per segment, with the wrappers
+stripped and the payload of a `bash -c` followed - are the ones the denylist
+uses now. A command that reaches the guard through a shape the kit does not
+know about is still let through, so this is a much taller fence, not a
+proof.
+
+## 7. New commands
+
+None of these replace anything. `oc-harness help` lists all of them.
+
+| Command | What it is for |
+|---|---|
+| `oc-harness doctor --watch` | Poll the catalogue and report only what changed. Free models are withdrawn in days, and a weekly check is the right rhythm. |
+| `oc-harness upgrade` | Check for a newer kit, refresh the profiles, and write the config again. |
+| `oc-harness cost` | What the kit is doing on THIS project, in numbers: characters pruned, tool outputs stubbed, lanes sorted by cost. |
+| `oc-harness backups`, `oc-harness restore` | The old config is now kept as a dated copy with the profile id in it, and it can be put back. |
+| `oc-harness uninstall` | Remove the symlinks the install created. `npm uninstall` leaves them behind, and opencode then loads neither the rules nor the guard. |
+| `oc-harness doctor --json` | The same report, machine-readable. |
+| `oc-harness show <id> --json` | The resolved profile, machine-readable. |
+
+Every command that can fail says so in its exit code, and
+[`docs/EXIT-CODES.md`](docs/EXIT-CODES.md) lists them. The four that matter
+most: `run` exits 1 when there is no task file, 2 when the work needs a
+person, 3 when the agent failed and 4 when the verify failed.
+
+## 8. The effort tier `"none"` means what it says
+
+A model that does not publish a `"none"` effort variant no longer gets one
+invented for it. It runs on its own default effort, and the config says
+which model served the request.
+
+## What did not change
+
+- The profile ids, the lane table, the plugin hook names (`guard`, `prune`,
+  `router`, `telemetry`), and the shape of `retention.yml`.
+- `oc-harness list`, `show`, `install`, `use`, `verify`, `refresh`, `baseline`,
+  `lanes` and `trust` keep their arguments.
+- The config format is unchanged. A config written by 0.3.x is read
+  correctly by 0.4.x; it is only missing the `kit` field.
+
+## If something looks wrong
+
+```bash
+oc-harness doctor      # what the kit sees, and what it would fix
+oc-harness verify      # whether the install matches the profile
+oc-harness use fleet-safe && oc-harness install && oc-harness verify
+```
+
+The last line rewrites the config from the profile and checks it. It is
+safe: the old config is written to `~/.config/opencode/backups/` first, and
+`oc-harness restore` puts it back.
+
+---
+
 # Migrating from 0.2.x to 0.3.0
 
 0.3.0 replaces the single-model profile with a **fleet**: ten jobs, ten free