@awebai/oats 0.35.0 → 0.35.1

package/docs/desktop.md

@@ -81,6 +81,9 @@ Desktop never parses the deployment: its members, lock state and header come
 from `oats workspace status`, and its instances from the deployment's one
 `agents/` root.
 Added workspaces are remembered and offered as suggestions next time.
+**Browse** opens in the folder holding the workspace most recently added or
+opened, or your home directory when there is none (never ~/Downloads);
+**Choose oats…** opens in the directory of the chosen or discovered CLI.
 
 Launch flags for scripted use: `--dir <workspace>` and `OATS_DESKTOP_PORT`.
 

package/docs/implementation.md

@@ -237,3 +237,20 @@ contacts GitHub, aweb, Jira or Linear.
 
 Tests pin behaviour, so a change that alters behaviour changes its test in the
 same commit. Never weaken an assertion to make a change pass.
+
+Since Git 2.47, a commit or fetch starts `git maintenance run --auto
+--detach`, a daemon that outlives the command. On a loaded CI runner it can
+still be repacking into a repository while the test's cleanup removes it,
+which fails with `ENOTEMPTY` (awebai/oats#451). So:
+
+- The shared fixture's repositories (`test/helpers/v2-deployment.mjs`: the
+  bare remote, the seed and the member clone) never run it. Each sets
+  `maintenance.auto=false` in its own config, which covers every git run in
+  it, a test's raw `execFileSync("git", …)` and the kernel's included.
+- A test that runs git itself goes through the fixture's `git()` helper,
+  which disables it per command, or sets `maintenance.auto=false` in a
+  repository it creates.
+- The kernel's remote cache passes `-c maintenance.auto=false` on every git
+  it runs there.
+
+Never paper over such a race with a retry around the cleanup.

package/docs/release-notes/v0.35.1.md

@@ -0,0 +1,42 @@
+# OATS 0.35.1
+
+## Fixed
+
+- **The remote cache no longer leaves a background git maintenance running
+  after a fetch** (awebai/oats#451). Since Git 2.47, a `git fetch` starts
+  `git maintenance run --auto --detach`, a daemon that outlives the fetch.
+  Every fetch the kernel made into its remote cache (`<cache>/<hash>`)
+  started one, which could still be repacking the cache after the command
+  that fetched had returned. The cache is the kernel's own repository and
+  never runs gc, so its git now runs with `maintenance.auto=false` as well.
+  The same daemon, started by a test's commit, was what made a retire test
+  fail its cleanup with `ENOTEMPTY` in CI; the test fixtures' repositories
+  turn it off too.
+- **Desktop's folder pickers open in a sensible place** (awebai/oats#460).
+  **Add workspace → Browse…** opened wherever macOS last was, often
+  ~/Downloads. It now opens in the folder holding the workspace most
+  recently added or opened, remembered across launches, and in the home
+  directory before the first one. **Choose oats…** opens in the directory
+  of the chosen or discovered CLI.
+
+- **A tree object the remote cache cannot read is a problem, never an empty
+  directory, and the cache heals** (awebai/oats#455). A fetch into the cache
+  that failed partway (on one host, one run by hand) left a member's commit
+  without its `souls/` tree. git answers "names no tree" the same way for an
+  absent path and for a tree object it cannot read, and the kernel took both
+  as an absent directory. So the member enumerated to zero souls with no
+  problem, the parsed cache (`<cache>/.parsed/`) kept that answer, and every
+  later command hid the member's souls (`E_SOUL_UNKNOWN`). Because a present
+  commit is never refetched, the cache stayed broken. Now:
+  - an absent directory is confirmed against the commit's own listing;
+  - a tree the commit lists but that cannot be read is `E_REMOTE_UNREADABLE`
+    (reason `cache`). It is shown as a problem and never kept in the parsed
+    cache;
+  - the read marks that cache repository damaged, and the next command
+    rebuilds it and fetches afresh.
+
+  Upgrading also stops a bad answer kept by an earlier kernel from being
+  served: the parsed cache is keyed by the kernel's own code.
+
+  Never fetch into `~/.cache/oats/remotes` by hand. It is the kernel's cache,
+  written only under its own locks.

package/lib/remote.mjs

@@ -696,7 +696,9 @@ function redactUrl(url) {
 }
 
 function repoHandle(dir, exec, session) {
-  const local = (args, opts = {}) => sessionExec(exec, session, ["-C", dir, "-c", "gc.auto=0", ...args], { cwd: dir, ...(session ? { signal: session.signal } : {}), ...opts });
+  // The cache is the kernel's own: no gc, and no `git maintenance run --auto --detach` (Git >= 2.47) left
+  // repacking it after the command that fetched has returned (awebai/oats#451).
+  const local = (args, opts = {}) => sessionExec(exec, session, ["-C", dir, "-c", "gc.auto=0", "-c", "maintenance.auto=false", ...args], { cwd: dir, ...(session ? { signal: session.signal } : {}), ...opts });
   return { dir, exec, local, session };
 }
 
@@ -781,6 +783,7 @@ class ReadSession {
     this.used = new Map();                 // memo key → { observedAt, reused }: the heads this command used
     this.peels = new Map();                // `${cacheRepo}\0${oid}` → peeled commit (positive answers only)
     this.trees = new Map();                // `${cacheRepo}\0${commit}` → Promise<tree index | null>
+    this.damaged = new Set();              // cache repos this session found damaged: rebuilt by the next one, not mid-command
     this.batches = new Map();              // cacheRepo → batch reader, in least-recently-used order
     this.retiring = new Set();             // evicted batch readers still ending
     this.parsedDirs = new Set();           // fingerprint dirs touched (mtime bumped) this session
@@ -1173,7 +1176,7 @@ async function ensureCommit(ref, oid, options) {
     // Already present (the common case): a read, so no write lock.
     let peeled = false; // the object store was asked already, and said no
     const usable = async (repo) => {
-      if (!existsSync(join(dir, "HEAD"))) return null;
+      if (!existsSync(join(dir, "HEAD")) || toRebuild(dir, session)) return null;
       if (!keepsPartialCache(await readVersion(repo, ref)) && await fetchMode(repo) === "partial") return null;
       peeled = true;
       return peelCommit(repo, oid);
@@ -1187,6 +1190,7 @@ async function ensureCommit(ref, oid, options) {
     const version = await readVersion(repo, ref);
     // A partial cache is only safe where git cannot fetch a missing blob on its own: an older git rebuilds it whole.
     if (!keepsPartialCache(version) && await fetchMode(repo) === "partial") repo = await rebuildWhole(repo, ref, options, version);
+    if (toRebuild(repo.dir, session)) repo = await freshCache(repo, ref, options);
     const cached = peeled && !waited ? null : await peelCommit(repo, oid);
     if (cached) return { repo, commit: remember(cached) }; // pinned AND present AND (peels to) a commit
     const mode = await fetchMode(repo)
@@ -1300,18 +1304,45 @@ export function keepsPartialCache(version) {
 /** Delete a partial cache an older git cannot read (it would fetch a missing blob on its own, or die), and start
  *  it again recording whole-tree fetches. The cache is disposable: everything in it is fetched again. */
 async function rebuildWhole(repo, ref, options, version) {
-  const session = repo.session;
-  if (session) {
-    session.batches.get(repo.dir)?.kill();
-    session.batches.delete(repo.dir);
-    for (const map of [session.peels, session.trees]) for (const k of [...map.keys()]) if (k.startsWith(`${repo.dir}\0`)) map.delete(k);
-  }
-  rmSync(repo.dir, { recursive: true, force: true });
-  const fresh = await cacheRepo(ref, options);
+  const fresh = await freshCache(repo, ref, options);
   await recordFullFetches(fresh, ref, olderGitNotice(version, ref));
   return fresh;
 }
 
+/** Drop a cache repository and start it afresh (under the cache's write lock); the session forgets it. */
+async function freshCache(repo, ref, options) {
+  forgetCacheRepo(repo);
+  rmSync(repo.dir, { recursive: true, force: true });
+  return cacheRepo(ref, options);
+}
+
+/** The session's peels, tree indexes and batch reader of this cache repository: what it read there is void. */
+function forgetCacheRepo(repo) {
+  const session = repo.session;
+  if (!session) return;
+  session.batches.get(repo.dir)?.kill();
+  session.batches.delete(repo.dir);
+  for (const map of [session.peels, session.trees]) for (const k of [...map.keys()]) if (k.startsWith(`${repo.dir}\0`)) map.delete(k);
+}
+
+/** A cache repository that holds a commit without a tree of it (a fetch into it that failed partway, say: a
+ *  present commit is never refetched) is marked damaged by the read that finds it, and the next ensureCommit
+ *  rebuilds it (awebai/oats#455). The mark lives in the repository, so it goes with it. The session that found
+ *  the damage keeps the repository as it is and reports problems: rebuilding it under that command's other
+ *  reads would remove what they are reading. The next session (or a read without one) rebuilds, inside
+ *  ensureCommit's cache write lock as every other cache write. The mark is a plain file in the bare
+ *  repository's root, which git ignores (as it does FETCH_HEAD or gc.log); writing it needs no lock: it only
+ *  asks for a rebuild, and a rebuild that already happened took the mark with it. */
+const DAMAGED_MARK = "oats-damaged";
+function markCacheDamaged(repo, why) {
+  try { writeFileSync(join(repo.dir, DAMAGED_MARK), `${why}\n`); } catch { /* the read's problem stands either way */ }
+  // The finding session keeps its reader, peels and tree indexes of the repository: they are still true, and
+  // freshCache forgets them when the repository actually goes.
+  repo.session?.damaged.add(repo.dir);
+}
+/** The repository carries a damage mark this session did not set: the next fetch rebuilds it first. */
+const toRebuild = (dir, session) => existsSync(join(dir, DAMAGED_MARK)) && !session?.damaged.has(dir);
+
 /** `git fetch` from the remote into the cache (args start at the subcommand and name the remote "origin"):
  *  a lost on-disk `.lock` race is retried, a timeout names the fetch (`what`) and how long it ran, any other
  *  failure is E_REMOTE_UNREADABLE (`raw`: the git error itself, for a caller that reads its stderr). → { stdout, stderr } */
@@ -1696,7 +1727,13 @@ function assertNoCollisions(entries, ref, commit, prefix) {
 /** `git ls-tree -l -z [flags] <spec> [-- <path>]`; null when <spec> names no tree.
  * Any other failure is E_REMOTE_UNREADABLE (reason "timeout" for the timeout kill, "killed"
  * for any other signal exit, else "unknown") — never a raw Node/git error: enumerateRepo turns E_REMOTE_* into a problem
- * row and would otherwise abort the whole discovery on one unexplained listing (L4). */
+ * row and would otherwise abort the whole discovery on one unexplained listing (L4).
+ *
+ * git answers "names no tree" the same way for a path that is absent and for a tree object it cannot read
+ * (`<commit>:souls` with that tree missing is "not a tree object"; a path below it is "not a valid object
+ * name"), so the message is never trusted alone: null only when the commit's own listing confirms the path is
+ * absent or not a tree (awebai/oats#455). A tree it lists but cannot read is E_REMOTE_UNREADABLE, which no
+ * parsed-cache item is ever written with: an absent directory would be cached as zero souls. */
 async function lsTree(repo, spec, { flags = [], path, ref, commit, maxBuffer = 64 * 1024 * 1024 } = {}) {
   try {
     const args = ["ls-tree", "-l", "-z", ...flags, spec];
@@ -1706,11 +1743,54 @@ async function lsTree(repo, spec, { flags = [], path, ref, commit, maxBuffer = 6
   } catch (error) {
     if (typeof error?.code === "string" && error.code.startsWith("E_")) throw error; // already an oats error
     const text = stderrText(error).toLowerCase();
-    if (/not a tree object|not a valid object name|does not exist|bad object|fatal: not a tree|path .* does not exist|exists on disk, but not in/.test(text)) return null;
-    const killed = !error?.timedOut && !error?.overflowed && typeof error?.signal === "string" && error.signal !== "";
-    const reason = error?.timedOut ? "timeout" : killed ? "killed" : "unknown";
-    const why = error?.overflowed ? "listing exceeded the output budget" : killed ? `git was killed (signal ${error.signal})` : (text.trim().split("\n")[0] || error?.code || error?.message || "git ls-tree failed");
-    throw fail("E_REMOTE_UNREADABLE", `cannot list ${spec}${path !== undefined ? ` -- ${path}` : ""} in ${ref?.key ?? repo.dir} (${reason}: ${why})`, { url: ref?.url ?? null, key: ref?.key ?? null, reason, commit: commit ?? null, spec, path: path ?? null, cause: error?.code ?? null, overflowed: error?.overflowed === true, ...(killed ? { signal: error.signal } : {}) });
+    const namesNoTree = /not a tree object|not a valid object name|does not exist|bad object|fatal: not a tree|path .* does not exist|exists on disk, but not in/.test(text);
+    if (namesNoTree) {
+      let absent;
+      // The confirming listing's own failure (a timeout, a kill, the session's abort) is classified as any listing
+      // failure is: it says nothing about the tree.
+      try { absent = await namesNoTreeIn(repo, spec); }
+      catch (confirmFailed) { throw listingUnreadable(confirmFailed, { repo, spec, path, ref, commit }); }
+      if (absent) return null;
+      const bare = !spec.includes(":");
+      markCacheDamaged(repo, bare ? `${spec}: the commit cannot be read` : `${spec}: listed by its commit, unreadable`);
+      throw fail("E_REMOTE_UNREADABLE", `cannot list ${spec} in ${ref?.key ?? repo.dir} (cache: ${bare ? "the commit's own tree cannot be read" : "its commit lists a tree there that cannot be read"})`, { url: ref?.url ?? null, key: ref?.key ?? null, reason: "cache", commit: commit ?? null, spec, path: path ?? null });
+    }
+    throw listingUnreadable(error, { repo, spec, path, ref, commit });
+  }
+}
+
+/** A failed `git ls-tree` as E_REMOTE_UNREADABLE: reason "timeout" for the timeout kill, "killed" for any other
+ *  signal exit, "cache" for an object the cache lacks (a tree below the commit that git cannot read: the
+ *  repository is marked damaged), else "unknown". An oats error passes through as itself. */
+function listingUnreadable(error, { repo, spec, path, ref, commit }) {
+  if (typeof error?.code === "string" && error.code.startsWith("E_")) return error;
+  const text = stderrText(error).toLowerCase();
+  const missingObject = /could not read [0-9a-f]{40,64}|missing (?:tree|blob|object) [0-9a-f]{40,64}|bad tree object/.test(text);
+  if (missingObject) markCacheDamaged(repo, `${spec}: ${text.trim().split("\n")[0]}`);
+  const killed = !error?.timedOut && !error?.overflowed && typeof error?.signal === "string" && error.signal !== "";
+  const reason = error?.timedOut ? "timeout" : killed ? "killed" : missingObject ? "cache" : "unknown";
+  const why = error?.overflowed ? "listing exceeded the output budget" : killed ? `git was killed (signal ${error.signal})` : (text.trim().split("\n")[0] || error?.code || error?.message || "git ls-tree failed");
+  return fail("E_REMOTE_UNREADABLE", `cannot list ${spec}${path !== undefined ? ` -- ${path}` : ""} in ${ref?.key ?? repo.dir} (${reason}: ${why})`, { url: ref?.url ?? null, key: ref?.key ?? null, reason, commit: commit ?? null, spec, path: path ?? null, cause: error?.code ?? null, overflowed: error?.overflowed === true, ...(killed ? { signal: error.signal } : {}) });
+}
+
+/** Whether `<commit>:<rel>` really names no tree: the commit's own listing of <rel>, which reads every tree on
+ *  the way, has no entry there, or a non-tree one. A bare commit spec, a tree entry, or a listing git ran and
+ *  failed means the tree is there and unreadable: false. A listing that did not run to its end (a timeout,
+ *  a kill, the session's abort, an oats error) answers nothing: it is thrown, for lsTree to classify. */
+async function namesNoTreeIn(repo, spec) {
+  const colon = spec.indexOf(":");
+  if (colon < 0) return false;
+  const rel = spec.slice(colon + 1);
+  if (!rel) return false;
+  try {
+    const out = await repo.local(["ls-tree", "-l", "-z", spec.slice(0, colon), "--", rel]);
+    const entry = parseLsTree(out.stdout).find((e) => e.path === rel);
+    return !entry || entry.type !== "tree";
+  } catch (error) {
+    const unfinished = error?.timedOut || error?.overflowed || (typeof error?.signal === "string" && error.signal !== "")
+      || error?.code === "ABORT_ERR" || (typeof error?.code === "string" && error.code.startsWith("E_"));
+    if (unfinished) throw error;
+    return false;
   }
 }
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@awebai/oats",
-  "version": "0.35.0",
+  "version": "0.35.1",
   "description": "OATS (Open Agent Team Specification) — durable souls, disposable instances, targetable capability packages, and the runtime-neutral oats CLI/kernel.",
   "keywords": [
     "agents",