@kaisers-io/refs 0.10.0 → 0.12.0

package/CHANGELOG.md

@@ -7,6 +7,401 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.12.0] - 2026-09-08
+
+### Upgrading
+
+Run `refs sync` after updating. Two things are worth expecting on that first run.
+
+**Packages may be reported as `missing` that are still on disk.** Negated workspace patterns are
+applied now, so a package a repository excludes (`!examples/vue/2*`) is no longer a workspace
+member — and an entry registered for one before this release no longer verifies. Nothing is removed
+automatically; the finding names the entry and leaves the decision alone.
+
+**Packages that arrived upstream since the last sync are reported.** Only those: a package the
+configuration never had and that did not arrive in the fetched range stays unmentioned, however
+long it has been there.
+
+### Added
+
+- **`refs sync` now reports a package that arrived upstream, and `refs edit --create` registers
+  it.** The drift probe checked the packages the configuration already had, so a package added
+  upstream after `refs add` stayed invisible — and there was no command to register one either:
+  `refs add` refuses an already-tracked ref, and every `refs edit` mode needs an entry to edit. The
+  only instruction anyone could give was "hand-edit `config.toml`".
+
+  `refs sync` answers "did upstream gain a package?" from the range it just fetched, not by
+  comparing a scan against the configuration. That distinction is the whole design: a scan cannot
+  tell a package that just arrived from one the ref's owner deliberately never tracked, because
+  there is no inventory of what was there before — the fetch range is that inventory. A ref whose
+  owner tracks 3 packages out of 140 hears about the other 137 exactly never. `refs doctor` lists
+  every unregistered member instead, because it was asked to.
+
+  The question it asks of that range is about package NAMES, not manifest paths. A package
+  renamed in place modifies its manifest rather than adding one, and a package merely moved to
+  another directory adds one without being new — so a path-based reading is wrong in both
+  directions. Only the manifests the range actually changed have to be read out of history: an
+  untouched manifest is byte-identical at both ends, so the name it carries now is the name it
+  carried before.
+
+  The repair is a command now rather than a config fragment, with the ref key filled in so it
+  runs as printed:
+
+  ```
+  refs edit 'github.com/acme/alpha' --package '@acme/new' --create --path 'packages/new' \
+    --description "<what it is>"
+  ```
+
+  It is a distinct mode, not an upsert — an ordinary field edit naming an unregistered package
+  still fails with `not_found`, so a typo in `--package` can never become a new entry. The
+  finding carries `name` and `path`, both verified against the checkout and both shell-quoted
+  (being verified makes a value true, not shell-safe: `zPackagePath` permits `$()` and a manifest
+  `name` is checked only for being non-empty), and deliberately no description: a manifest description is untrusted third-party content, and copying it moves it
+  into a file refs later reads as its own configuration. The skill instructs agents to propose the
+  registration and wait for the user to agree, rather than run it on their own initiative.
+
+### Fixed
+
+- **Workspace patterns are matched by `minimatch`, the matcher npm itself uses.** Hand-written
+  matching disagreed with the real resolvers in five distinct ways, each found only after the last
+  was fixed: extglob (`@(a|b)`) read as a literal directory name, trailing slashes treated
+  symmetrically where `minimatch` is asymmetric, repeated separators silently matching nothing, and
+  two more. Delegating removes that class of defect rather than the current instance of it.
+
+  Measured before choosing: `picomatch` disagrees with `minimatch` on ten of 154 comparisons over
+  the shapes this scanner supports, exactly on trailing slashes and repeated separators — so it is
+  not a drop-in. pnpm matches through `picomatch` but normalizes first, and was measured to agree
+  with `minimatch` on every one of those shapes, so one matcher covers both ecosystems.
+
+  Walking stays here: containment guards, the diagnostics that say why a scan came up short, and
+  the deliberate one-level depth policy are unchanged. `minimatch` answers only whether a path
+  matches a pattern. Nothing new is installed by `refs` users — the CLI publishes a bundle with no
+  dependencies — and that bundle grows by 24 KB.
+
+  One behaviour improves as a consequence: a negation in a shape this scanner cannot WALK
+  (`!packages/{a,b}`) is now applied, because applying an exclusion needs matching and never
+  walking. It used to be reported as unsupported and silently ignored.
+
+- **Negated workspace patterns are applied instead of ignored.** `!packages/fixtures` was dropped
+  as an unsupported shape (a v1 simplification), so `refs add` registered packages the repository
+  had explicitly excluded, and every finding about a repository declaring one was silenced —
+  TanStack Query declares two, and all hundred of its packages came back `unverifiable`. Negations
+  are now expanded exactly like inclusive patterns and subtracted from the result, which makes the
+  scan an accurate statement of membership rather than an approximation of one.
+
+  A wildcard inside the last segment (`examples/vue/2*`) is supported too, since that is the shape
+  real repositories exclude by, and a negation nobody can expand costs every finding about the
+  repository rather than just the paths it names. Glob syntax that is still unimplemented —
+  `{a,b}`, `?`, `[…]` — now reports `unsupported_pattern` rather than reading as a literal
+  directory name and silently matching nothing.
+
+- **A monorepo can now be resolved by the name in its own root manifest.** Workspace detection
+  expands the globs a repository declares, and a workspace root is not one of its own targets — so a
+  root that names itself was registered nowhere, and `refs resolve @acme/toolkit` came back empty
+  for a repository that was tracked all along. `refs add` now registers a named root at `path: "."`
+  alongside the workspace members.
+
+  Both pnpm and Yarn address a workspace root by that name (`pnpm --filter <root-name>`,
+  `yarn workspace <root-name>`); npm and Turborepo use a positional handle instead. Of eighteen
+  well-known monorepos surveyed, eight carry a name someone would plausibly use for the repository
+  and ten carry a throwaway like `root` or `monorepo-root` — which is what settles it: registering
+  the name costs nothing where it is a throwaway, since nobody resolves `"root"`, and answers the
+  question where it is not.
+
+  Two things this deliberately does not do. A repository that declares no workspaces is untouched:
+  `refs add`'s npm fallback owns that shape, and probing the root there would displace a locator it
+  did not choose — as it also would where a workspace declaration selects nothing, so the package
+  named in an `npm:<pkg>` source survives there too. And where a workspace member already claims the
+  root's name — `@remix-run/react-router` is a real example, in a repository that also publishes
+  `react-router` — the member wins and the root is simply not registered, which costs that
+  repository nothing it had before. That rule lives in detection itself rather than in `refs add`,
+  so relocation agrees with registration: a member that moves is still found uniquely, instead of
+  becoming ambiguous against a same-named root and leaving `resolve` with no path for a package
+  that is plainly there. And a root is never reported as a package's new location: its name is an
+  alias for the repository, so a member that upstream deletes is reported as gone rather than as
+  having moved to the repository root — which would have sent a caller to the wrong directory and
+  described a move that never happened.
+
+  Refs tracked before this change keep the package map they were given, and no command adds one
+  entry to an existing ref — `refs add` refuses a tracked ref, `refs edit --package` needs an entry
+  to edit. So `refs sync` and `refs doctor`'s `config-drift` check now report a root the
+  configuration does not register, with the entry to add — including the path registration would
+  actually use, which is the member's rather than the root's where a workspace member declares the
+  same name. That costs one manifest read per ref, and a workspace scan only where there is
+  something to report; it is asked only of refs that already register packages, since a plain
+  reference repository registers none on purpose and is left alone.
+
+  The root package takes the ref's own description when its manifest carries none, which is the
+  ordinary case for a private workspace root. That is not the per-package fallback `refs add`
+  otherwise refuses: the root is not a package beside the repository, it is that repository.
+
+- **A failed lookup no longer reads as an absent repository.** `refs resolve` exits `4` when a query
+  matches nothing, and the message ended "run refs list, or add it: `refs add <url>`". That second
+  half is a guess: a query can miss every route while the repository is tracked perfectly well under
+  another identifier — a monorepo root whose own package name was never registered, for instance.
+  An agent read the suggestion as confirmation and told someone a repository they had tracked was
+  not tracked, then stopped.
+
+  The message now states what was searched and points at evidence rather than prescribing a fix, and
+  `--json` carries a `reason` on `resolve`'s routing misses: `unmatched_query` (nothing matched, by
+  any route), `package_not_registered` (the ref is tracked and registers no such package), or
+  `ref_not_registered` (a canonical git url named an absent ref — the one case where adding it is
+  the right answer, since only a canonical url establishes which ref was meant). There is deliberately no reason meaning "this repository does not exist", because nothing
+  refs can observe establishes that; and `reason` is absent on every other `not_found`, where its
+  absence means no narrowing is available rather than being a fourth value.
+
+  The skill's instruction changed with it. It used to say exit `4` means the ref is not tracked; it
+  now says exit `4` means the query matched no route, and requires a second lookup before any
+  conclusion. `refs resolve --ref <ref>`'s own miss also stopped suggesting a bare `refs show`,
+  which reports a package count and no names — it now suggests `--packages`, which actually shows
+  the map the reader was sent to inspect.
+
+- **A stale-lock reclaim could delete a lock another process was using.** refs reclaims a lock left
+  behind by a crashed process. The check that decided a lock was abandoned and the removal that
+  acted on it were two separate steps, and in the gap between them the lock could legitimately
+  become somebody else's: the original holder releases, a waiting process takes the same path, and
+  the reclaim then deletes a lock that is actively in use. Both processes go on to `reset --hard`
+  the same checkout.
+
+  Three changes close it for processes running this version:
+
+  - **Only a process the operating system reports as gone is reclaimed from automatically.** A lock
+    whose lease has run out but whose process still answers is now reported rather than taken — a
+    live process can release at any instant, and that release is what opened the gap. Same for a
+    lock whose metadata never finished being written, or carries no usable identity.
+  - **The acquisition is re-identified after the death check.** Proving the recorded process gone is
+    not enough on its own: the metadata is read first and the process probed after, so the path can
+    change hands in between and the probe then answers about the departed owner. Re-reading the
+    identity immediately before the removal is what ties the two together.
+  - **The marker that stops two reclaims colliding no longer expires.** It used to be taken over
+    after two seconds, so it only excluded a reclaim fast enough to finish inside that window; a
+    suspended one lost its marker mid-work and a second reclaim started on the same lock. Age is
+    not evidence of abandonment.
+
+  The protocol's own markers also moved into `locks/.claims/` and `locks/.tombstones/`, which no
+  lock name can reach — lock names must start with a letter or digit. That removes a collision
+  where a repository literally named `foo.steal-claim` produced the marker path of the lock for
+  `foo`, and with it the name-shape guessing `refs doctor` needed to tell the two apart.
+
+  **What this costs.** Three situations no longer recover on their own and need one explicit
+  command, which `refs doctor` prints along with the condition for running it safely — stop every
+  refs process on that home first, suspended ones included: a crash after the operating system has
+  reused the process id, a crash before the lock finished writing its metadata, and a crash while a
+  reclaim was starting.
+
+  The messages changed to match. `refs doctor`'s `locks` check separates a lock refs will reclaim
+  by itself from one it will not, and the failure message when a lock cannot be acquired no longer
+  says "already reclaimable — retry" for a lock nothing will ever reclaim. It also stopped
+  describing the window as the thing that frees a lock: waiting does not, and only a recorded
+  process the operating system reports as gone does.
+
+  **What it does not fix.** A refs process running an _older_ version follows none of this and
+  reclaims on its own terms. And no lock protocol can help when refs is hard-killed while its `git`
+  child survives: the successor's lock is honest about the lock, not about the directory.
+
+- **Two unrelated refs could share one lock.** The per-ref lock name replaced `/` with `_`, and `_`
+  is legal inside a ref key — so `github.com/acme_tools/widget` and `github.com/acme/tools_widget`
+  both derived `ref.github.com_acme_tools_widget`. The two then serialized against each other:
+  `sync` fans out four refs at a time and the loser failed on a lock conflict after the timeout,
+  `resolve`'s verification could block on a sync of a ref it has nothing to do with, and `doctor`'s
+  `config-drift` check reported the wrong ref as busy.
+
+  The name is now injective. A key containing no `_` encodes exactly as before, so it keeps the
+  lock name refs has always written for it; a key containing one moves into an escaped form under
+  `ref._`, where `_` becomes `_u` and `/` becomes `_s`. The two forms cannot collide, because a ref
+  key always starts with `[a-z0-9]` and so a plain name never begins `ref._`.
+
+  A lock name is one directory entry, so a key long enough to overflow one now falls back to a
+  digest under `ref.__` rather than failing `mkdir` with `ENAMETOOLONG`. The budget reserves room
+  for the sibling entries the steal protocol derives from a lock name — a name that could be
+  created but not renamed to its tombstone would strand an abandoned lock that nothing could then
+  reclaim, leaving the ref blocked until someone deleted the directory by hand. Both were already
+  true before this change, for keys past roughly 200 characters; reaching it needs a self-hosted
+  url, since no forge allows a path that long.
+
+  **One caveat if you run refs concurrently across an upgrade.** The lock name changes for a ref
+  whose key contains `_`, and for one long enough to reach the digest form — roughly 200
+  characters, either way. For such a ref, a process from any earlier release (the old scheme dates
+  to 0.1.1) derives a different name from a new one, so for the length of that overlap the two
+  would not exclude each other. The window is a mid-upgrade concurrent run on the same refs home;
+  if that is a situation you can be in, let running operations finish before upgrading. Every other
+  ref keeps its name and is unaffected.
+
+## [0.11.0] - 2026-08-31
+
+### Changed
+
+- **The skill's version-question flow moved into its own file.** Every question about a
+  dependency's source loads `SKILL.md` and `INVESTIGATE.md` in full, and 30 % of
+  `INVESTIGATE.md` was a block on resolving versions to tags and diffing between them — read
+  on every plain source question, used on almost none of them. It is now `VERSIONS.md`, with a
+  route of its own in `SKILL.md` §5 and a one-line pointer at the end of `INVESTIGATE.md`, so
+  a mis-route costs one extra read rather than a wrong answer.
+
+  Alongside it, a compression pass over what remains: the worker output contract was stated
+  twice and is now stated once, the five clickable-link rules became one normalization, and a
+  handful of sentences that repeated something said a few lines earlier are gone. Nothing
+  behavioural was removed — the capability gate, the trust boundary, the five hard rules and
+  the worker prompt's own safety rules are untouched, and the measured partial-clone cost model
+  (which git commands fetch blobs, and that `git blame` fetches one per visited revision) is
+  refs' own measurement rather than something the term "partial clone" implies, so it stays
+  verbatim.
+
+  A plain source question now loads 20 % less. A version question loads about what it did
+  before, plus one extra file read.
+
+### Added
+
+- **`refs sync` and `refs doctor` now report configuration that has fallen behind its upstream.**
+  A configured package path is only a locator, and upstream can delete or move what sits at it.
+  Until now only `refs resolve` noticed, for the one package an agent happened to route to, and it
+  persisted nothing — so a package deleted upstream could sit wrong in the configuration
+  indefinitely while every other package in the same checkout went uninspected.
+
+  Each successful `refs sync` result now carries a nested `structure: {status, packages}`, probed
+  inside the lock the sync already holds, right after the checkout was updated. Nothing is stored:
+  the answer is reported and thrown away, so there is no drift state that can itself go stale. A
+  removal and a relocation are reported as different findings, because they need opposite repairs —
+  telling an agent to "fix the path" of a package upstream deleted sends it looking for something
+  that is not there. Human output gains indented lines under the affected ref and stays silent when
+  everything resolves; the summary counts and exit code are untouched, since a drifted ref synced
+  perfectly well.
+
+  Only refs that actually sync are probed, which keeps `--stale-only` a genuine no-op — and is why
+  `refs doctor` gains a `config-drift` check as the deliberate "check everything now" counterpart.
+  It takes each ref's lock with a short timeout and reports the ref as busy rather than waiting,
+  writes nothing, and reports `warn` rather than `fail`: the configuration has fallen behind,
+  nothing is broken. `refs list` deliberately stays blind — without stored state it would
+  turn a cheap inventory command into a locking filesystem sweep.
+
+- **`refs resolve` answers in one call what used to take three.** The skill's investigation flow
+  began `resolve` → `sync` → `resolve` **again**, and the third call was not ceremony: package
+  verification had described the checkout as it was _before_ the sync, so reusing that answer meant
+  reporting a path that no longer necessarily held what it claimed. `--sync-if-stale` fetches (or
+  clones) only when the ref is stale or its checkout absent, and everything it reports describes the
+  checkout afterwards. The rule has left the skill and become code.
+
+  It refuses, rather than syncing, when the checkout is `unmanaged` or `unverifiable`. `sync`
+  hard-resets and cleans; running it against a directory whose identity was never established is
+  how a stray clone loses its history. A failing sync fails the command rather than returning a
+  success envelope containing a stale path.
+
+- **`refs resolve --project <dir>` reports the version a project has installed.** The skill used to
+  tell the agent to read the project's lockfile by hand, and nothing in refs touched one — so the
+  deterministic half of every "what changed between my version and a newer one" question was done
+  by the least deterministic component available, against pnpm's peer-qualified keys, aliases,
+  overrides and three vendor-specific formats.
+
+  The answer is read from `node_modules`, walking up in Node's own lookup order, and stops at the
+  first installation slot that exists rather than the first readable manifest — falling through to
+  an ancestor would report a shadowed install Node would not have loaded. There is deliberately no
+  lockfile fallback: a lockfile says what _should_ be installed, `node_modules` says what _is_, and
+  the second is the question. `installed.status` is `found`, `not_materialized`,
+  `unsupported_layout` (Yarn PnP, detected but never loaded — `.pnp.cjs` is project code) or
+  `unverifiable`.
+
+- **`refs resolve --ref <ref>`** scopes a query to one ref's packages. A package name registered by
+  several refs used to be answered with "use the full ref key" — advice the command could not
+  honour, because a full-key query routes by _ref_ and comes back with `package: null`. The error
+  now names a remedy that exists.
+
+- `refs doctor` gained a `locks` check. A held lock used to be invisible: acquisition failed with a
+  message that named no owner, and `doctor` had no lock check at all — so the one command meant to
+  answer "is something stuck?" could not see the thing that was stuck. The check lists every entry
+  in the locks directory with its recorded owner, how long it has been held, and against which
+  window.
+
+  A held lock is **not** a warning by itself: that is what a concurrent `refs sync` looks like, so
+  it reports `ok` with the holder listed. `warn` is reserved for something that will not resolve on
+  its own — a recorded process that is gone, a lock past its window and still there, metadata that
+  cannot be read, or something that is not a lock at all occupying a lock name. Like every other
+  `doctor` warning, it does not change the exit code.
+
+### Changed
+
+- **`refs resolve` establishes that the path it hands back is really this ref's checkout.** It used
+  to report presence from a `.git` entry alone, while `add` and `sync` both ran a stronger guard
+  before mutating — so the one command whose result is read as "the source is here" was the one
+  that did not check what was there. A manual clone at the derived path, a half-finished `remove`,
+  a restored backup or a symlinked second home all produced a confident answer about the wrong
+  repository, with no error and no warning.
+
+  Every reply now carries `checkout: {status, reason?}` — `managed`, `missing`, `unmanaged` or
+  `unverifiable` — read straight out of `.git/config` without spawning git, so the hot path stays
+  subprocess-free. The origin URL is never echoed back in `reason`; it can carry credentials.
+
+  `managed` requires the `core.hooksPath` marker to be **this home's** hooks directory, not merely
+  present — the comparison `add` already makes — so a manual clone that sets it for its own purposes
+  does not pass. A config git itself would reject (an unterminated quote, an undefined escape, a
+  line that is neither a section nor an assignment) is `unverifiable` rather than partially read: a
+  file git would not accept is not evidence of identity.
+
+  **Package verification is gated on it.** A manifest read inside an unrelated checkout can answer
+  `verified` for a package that has nothing to do with the query, so anything other than `managed`
+  or `missing` now yields `package.status: "unverifiable"` instead of a confident location.
+
+  `missing` is unchanged and still means `checkout.status === "missing"`. Callers should branch on
+  `checkout.status`, which answers the question `missing` was often assumed to.
+
+- **The "lock is held" error now says who holds it and for how long.** It used to read `lock <name>
+is held — another refs process is running`, which left no way to tell a running `sync` from
+  something that crashed 90 seconds ago. It now names the recorded pid, whether that pid is still
+  present, how long the lock has been held, and when it becomes reclaimable.
+
+  It deliberately says "recorded pid … is present (identity not verified)" rather than "held by pid
+  …": only `ESRCH` establishes that a process is gone, so a pid that answers may equally be an
+  unrelated process that reused the number. And it says "reclaimable", never "released
+  automatically" — nothing removes a lock in the background; the phrase means the next acquisition
+  attempt is entitled to take it.
+
+- **An operation that ran without the lock it asked for now fails instead of reporting success.**
+  A lock can still be lost while its holder works — a stolen lock is detected by the next renewal,
+  or by release finding a foreign token. Previously the callback's result was returned as if
+  nothing had happened. It is now reported as a `conflict` (exit code 5), because the work ran
+  without the mutual exclusion it requested and its result is not trustworthy. An operation that
+  failed on its own keeps precedence: its own error is what the caller sees.
+
+### Fixed
+
+- The `rm -rf` commands `refs doctor` and `refs add` suggest are now quoted. Ref keys derive from
+  user-supplied urls and permit spaces, `$()`, backticks, semicolons and quotes, and the refs home
+  itself routinely sits under a path containing a space — so pasting an unquoted suggestion could
+  delete several wrong paths and leave the intended one, or execute a command substitution embedded
+  in a repository name. The form is now `rm -rf -- '<path>'`; `--` additionally stops a path
+  beginning with `-` from parsing as options.
+
+- A pid in a lock's metadata is now required to be a positive integer within the range
+  `process.kill` accepts. `0` and negative values are
+  process-_group_ selectors for `process.kill`, so metadata carrying either made the liveness probe
+  answer for a whole group — reporting a long-gone owner as present, and keeping its lock
+  unreclaimable for the rest of its window. A value past that range is worse still: Node rejects it
+  with a `TypeError` rather than an errno, which the probe read as "not gone, therefore present".
+  Such metadata is now reported as malformed instead of acted on.
+
+- A lock is no longer taken away from a holder that is still working, however long the work takes.
+  Locks were judged by a fixed ten-minute age: past it, a waiter treated the lock as abandoned and
+  stole it even when its owner was demonstrably alive and mid-operation. Since the per-ref lock is held across a whole clone or
+  fetch, any repository large enough to take ten minutes could end up with two processes running
+  `checkout -B` / `reset --hard` / `clean -fd` against the same directory.
+
+  A holder now renews a lease while it works, so a lock is abandoned when its process is definitely
+  gone **or** its lease has expired — never merely because the work took a long time. "The pid still
+  exists" does not override an expired lease, which is what keeps a recycled pid from stranding a
+  lock forever.
+
+  The same constant was also too long at the other end: a crashed `sync` or `add` left its lock
+  behind and blocked the ref for the full ten minutes. An abandoned lock is now reclaimable after
+  two minutes rather than ten.
+
+  Two limits are worth knowing. A holder whose event loop is stopped for longer than the lease — a
+  suspended process, a sleeping machine — cannot renew, and is stealable after two minutes where it
+  used to take ten; in practice a sleeping machine suspends every refs process on it, and the
+  pending renewal fires on resume. And a lock written by an older CLI carries no lease and is still
+  judged by the ten-minute rule it was written under, so upgrading never dispossesses a running
+  older process. The reverse does not hold:
+  an older CLI reads no lease, so it can still take a lock from a live current holder once ten
+  minutes pass. Holds longer than that during a rolling upgrade are not protected.
+
 ## [0.10.0] - 2026-08-13
 
 ### Added
@@ -514,7 +909,9 @@ trusted-publishing pipeline end to end.
   installed git hooks.
 - Agent skill (`skills/refs/`) documenting the investigate/add/maintain workflows.
 
-[Unreleased]: https://github.com/kaisers-io/refs/compare/v0.10.0...HEAD
+[Unreleased]: https://github.com/kaisers-io/refs/compare/v0.12.0...HEAD
+[0.12.0]: https://github.com/kaisers-io/refs/compare/v0.11.0...v0.12.0
+[0.11.0]: https://github.com/kaisers-io/refs/compare/v0.10.0...v0.11.0
 [0.10.0]: https://github.com/kaisers-io/refs/compare/v0.9.0...v0.10.0
 [0.9.0]: https://github.com/kaisers-io/refs/compare/v0.8.3...v0.9.0
 [0.8.3]: https://github.com/kaisers-io/refs/compare/v0.8.2...v0.8.3