next-issue 2.18.0 → 2.20.0

package/README.md

@@ -304,12 +304,13 @@ Three levels of console output:
 - Node 24 or later
 - `gh`, authenticated with write access to the repository
 - pi, logged in to a model provider. The agents use the model, the thinking
-  level and the extensions from your pi settings, except the `pi-next-issue`
-  extension. The implementer and the fixer get the default tools of your pi
-  settings, the `grep`, `find` and `ls` tools and the tools of your
-  extensions. The reviewer gets all of them except `edit` and `write`. The
-  agents run with `GIT_OPTIONAL_LOCKS=0`, so a `git status` from an agent or
-  an extension does not hold the index lock while the harness commits.
+  level, the extensions and the skills from your pi settings, except the
+  extension and the skills of `pi-next-issue`. The implementer and the fixer
+  get the default tools of your pi settings, the `grep`, `find` and `ls` tools
+  and the tools of your extensions. The reviewer gets all of them except
+  `edit` and `write`. The agents run with `GIT_OPTIONAL_LOCKS=0`, so a
+  `git status` from an agent or an extension does not hold the index lock
+  while the harness commits.
 
 ## Configuration
 
@@ -451,8 +452,8 @@ state of an issue that a label still holds back stays as it was.
 | Argument    | Effect                                                                                                                        |
 | ----------- | ----------------------------------------------------------------------------------------------------------------------------- |
 | `init`      | Write the config file with the defaults                                                                                       |
-| `serve`     | Run the worker that pi extensions talk to, see [Worker](#worker)                                                              |
-| `stop`      | Stop the worker, see [Worker](#worker)                                                                                        |
+| `serve`     | Run the worker of this repository that pi extensions talk to, see [Worker](#worker)                                           |
+| `stop`      | Stop the worker of this repository, see [Worker](#worker)                                                                     |
 | `logs`      | Show the latest run log and follow it, see [Observability](#observability)                                                    |
 | `--force`   | Replace an existing config file, with `init`                                                                                  |
 | `--once`    | Stop after the first handled issue                                                                                            |
@@ -498,20 +499,20 @@ The watch stops:
 
 ## Worker
 
-`next-issue serve` runs one worker for the machine. The
+`next-issue serve` runs one worker for the repository it is run in. The
 [`@ravshansbox/pi-next-issue`](../pi-next-issue) extension starts it and talks
-to it. The worker:
-
-- listens with WebSocket on the socket file `~/.pi/agent/next-issue.sock`, or
-  in the folder that `PI_CODING_AGENT_DIR` names. Only your user can open the
-  file. It works on macOS and Linux;
-- holds `next-issue.lock` in the same folder, with its process id, so only one
-  worker runs. A lock whose process is no longer alive is taken over;
-- keeps the repositories it knows in `next-issue.roots.json` in the same
-  folder, so it still knows them after a restart;
-- runs at most one run per repository, and runs in several repositories at
-  once. Each run is like `next-issue` without `--watch`, and the client says
-  whether it merges;
+to it. Each repository has its own worker, so the runs, restarts and upgrades
+of one repository never affect another. The worker:
+
+- listens with WebSocket on the socket file `.next-issue/worker.sock` in the
+  repository. Only your user can open the file. It works on macOS and Linux.
+  When the path of the socket file is too long for the system, about 103
+  bytes on macOS and 107 on Linux, it exits with an error that says so;
+- holds `.next-issue/worker.lock` in the repository, with its process id, so
+  only one worker runs per repository. A lock whose process is no longer alive
+  is taken over;
+- runs at most one run at a time. Each run is like `next-issue` without
+  `--watch`, and the client says whether it merges;
 - sends every client the progress of every run when it connects and on every
   change: for each running issue the issue, the step, the tool count and the
   last tool, and the handled issues;
@@ -521,7 +522,7 @@ to it. The worker:
   repository, the pull request when there is one, and the `needsHuman` and
   `skip` labels. It reads them
   from GitHub, so the list is the same after a restart. It sends them when a
-  client connects, for each repository it knows, after each issue of a run
+  client connects, for its repository, after each issue of a run
   ends, and when a client sends `{ "type": "questions", "root": "<path>" }`.
   An issue that no longer has the label leaves the list. An issue with the
   label but no hand-over comment is in the list with an empty reason and url.
@@ -531,17 +532,34 @@ to it. The worker:
   `{ "type": "questions-error", "root": "<path>", "message": "<text>" }`;
 - stops a run, when a client asks, like a second `Ctrl-C`: each running issue
   after its current round, with the state kept;
-- stays when all clients close;
+- exits, and removes its socket and lock files, when no run is going and no
+  client is connected, or when no client connects in the 30 seconds after it
+  starts;
+- waits up to 5 seconds for the lock when a worker that is closing still
+  holds it;
 - closes once no run is going when a client with a newer version connects, so
   that the client can start the newer worker;
 - stops every agent and command and exits on `SIGTERM` or `SIGINT`, and
   removes its socket and lock files.
 
-Stop it with `next-issue stop`. This sends `SIGTERM` to the process in the
-lock file, so the worker stops at once and a run that is going resumes from
+Stop the worker of a repository with `next-issue stop` in that repository.
+Workers of other repositories carry on. This sends `SIGTERM` to the process in
+the lock file, so the worker stops at once and a run that is going resumes from
 its saved state next time. When no worker is running, the command says so and
 exits with 0.
 
+Versions before one worker per repository ran one worker for the machine,
+with `next-issue.sock`, `next-issue.lock` and `next-issue.roots.json` in
+`~/.pi/agent`, or in the folder that `PI_CODING_AGENT_DIR` names. That worker
+does not stop by itself, `next-issue stop` does not reach it, and a run in it
+can claim issues alongside the new worker of the same repository. Stop it once
+after you upgrade:
+
+```sh
+kill "$(cat ~/.pi/agent/next-issue.lock)"
+rm -f ~/.pi/agent/next-issue.roots.json
+```
+
 ## Development
 
 ```bash

package/dist/agents.mjs

@@ -9,7 +9,7 @@ const CODING = {
     tools: ['+grep', '+find', '+ls'],
 };
 export { CODING, REVIEWING };
-export function isOwnExtension(path) {
+export function isOwnResource(path) {
     return path.split(sep).includes('pi-next-issue');
 }
 export async function runAgent(recorder, request) {
@@ -24,7 +24,11 @@ export async function runAgent(recorder, request) {
         agentDir,
         extensionsOverride: (base) => ({
             ...base,
-            extensions: base.extensions.filter((extension) => !isOwnExtension(extension.resolvedPath)),
+            extensions: base.extensions.filter((extension) => !isOwnResource(extension.resolvedPath)),
+        }),
+        skillsOverride: (base) => ({
+            ...base,
+            skills: base.skills.filter((skill) => !isOwnResource(skill.filePath)),
         }),
     });
     await resourceLoader.reload();

package/dist/args.mjs

@@ -2,8 +2,8 @@ import { CONFIG_FILE } from './config.mjs';
 export const USAGE = `next-issue [command] [options]
 
   init         write ${CONFIG_FILE} with the defaults
-  serve        run the worker that pi extensions talk to
-  stop         stop the worker
+  serve        run the worker of this repository that pi extensions talk to
+  stop         stop the worker of this repository
   logs         show the latest run log and follow it
 
   --once       stop after the first handled issue

package/dist/cli.mjs

@@ -6,7 +6,7 @@ import { listenForInterrupts } from './interrupt.mjs';
 import { followLog, newestRunLog } from './logs.mjs';
 import { message } from './observe.mjs';
 import { runRepo } from './run.mjs';
-import { serve, socketPaths, stopWorker } from './serve.mjs';
+import { serve, stopWorker, workerPaths } from './serve.mjs';
 import { STATE_DIR } from './state.mjs';
 import { formatSummary } from './summary.mjs';
 function write(stream, text) {
@@ -20,17 +20,17 @@ async function main() {
         await write(process.stdout, USAGE);
         return 0;
     }
+    const root = await repoRoot(process.cwd());
     if (args.command === 'serve') {
-        return serve();
+        return serve(root);
     }
     if (args.command === 'stop') {
-        const pid = await stopWorker(socketPaths().lockPath);
+        const pid = await stopWorker(workerPaths(root).lockPath);
         await write(process.stdout, pid === undefined
             ? 'next-issue: no worker is running\n'
             : `next-issue: stopped the worker with process id ${pid}\n`);
         return 0;
     }
-    const root = await repoRoot(process.cwd());
     if (args.command === 'init') {
         return init(root, args.force);
     }

package/dist/serve.mjs

@@ -1,8 +1,7 @@
 import { rmSync } from 'node:fs';
-import { readFile, rm, writeFile } from 'node:fs/promises';
+import { mkdir, readFile, rm, writeFile } from 'node:fs/promises';
 import { createServer } from 'node:http';
-import { join } from 'node:path';
-import { getAgentDir } from '@earendil-works/pi-coding-agent';
+import { dirname, join } from 'node:path';
 import { WebSocketServer } from 'ws';
 import { parseArgs } from './args.mjs';
 import { loadConfig } from './config.mjs';
@@ -14,14 +13,24 @@ import { applyEvent, startProgress } from './progress.mjs';
 import { PORTS } from './ports.mjs';
 import { isNewer, } from './protocol.mjs';
 import { runRepo } from './run.mjs';
+import { STATE_DIR } from './state.mjs';
 import { packageVersion } from './version.mjs';
-export function socketPaths(agentDir = getAgentDir()) {
+const LOCK_WAIT_MS = 5_000;
+const LOCK_POLL_MS = 100;
+const IDLE_MS = 30_000;
+export function workerPaths(root) {
     return {
-        socketPath: join(agentDir, 'next-issue.sock'),
-        lockPath: join(agentDir, 'next-issue.lock'),
-        rootsPath: join(agentDir, 'next-issue.roots.json'),
+        socketPath: join(root, STATE_DIR, 'worker.sock'),
+        lockPath: join(root, STATE_DIR, 'worker.lock'),
     };
 }
+export function checkSocketPath(path, platform = process.platform) {
+    const size = platform === 'linux' ? 108 : 104;
+    const length = Buffer.byteLength(path);
+    if (length >= size) {
+        throw new Error(`the socket path ${path} is too long for this system: ${length} bytes, at most ${size - 1}`);
+    }
+}
 function alive(pid) {
     try {
         process.kill(pid, 0);
@@ -42,8 +51,9 @@ export async function stopWorker(lockPath) {
     }
     return pid;
 }
-async function lock(path) {
-    for (let attempt = 0; attempt < 2; attempt += 1) {
+async function lock(path, waitMs) {
+    const deadline = Date.now() + waitMs;
+    for (;;) {
         try {
             await writeFile(path, `${process.pid}\n`, { flag: 'wx', mode: 0o600 });
             return true;
@@ -52,21 +62,16 @@ async function lock(path) {
             if (error.code !== 'EEXIST') {
                 throw error;
             }
-            if ((await holder(path)) !== undefined) {
-                return false;
-            }
+        }
+        if ((await holder(path)) === undefined) {
             await rm(path, { force: true });
         }
-    }
-    return false;
-}
-async function readRoots(path) {
-    try {
-        const value = JSON.parse(await readFile(path, 'utf8'));
-        return new Set(Array.isArray(value) ? value.filter((root) => typeof root === 'string') : []);
-    }
-    catch {
-        return new Set();
+        else if (Date.now() >= deadline) {
+            return false;
+        }
+        else {
+            await new Promise((resolve) => setTimeout(resolve, LOCK_POLL_MS));
+        }
     }
 }
 export async function listQuestions(root) {
@@ -74,23 +79,13 @@ export async function listQuestions(root) {
     return waitingQuestions(await detectRepo(root, config.remote), config, PORTS);
 }
 export async function startWorker(options) {
-    if (!(await lock(options.lockPath))) {
+    checkSocketPath(options.socketPath);
+    await mkdir(dirname(options.lockPath), { recursive: true });
+    if (!(await lock(options.lockPath, options.lockWaitMs ?? LOCK_WAIT_MS))) {
         return undefined;
     }
     await rm(options.socketPath, { force: true });
     const runs = new Map();
-    const roots = await readRoots(options.rootsPath);
-    let saving = Promise.resolve();
-    const remember = (root) => {
-        if (roots.has(root)) {
-            return;
-        }
-        roots.add(root);
-        const value = `${JSON.stringify([...roots], null, 2)}\n`;
-        saving = saving
-            .then(() => writeFile(options.rootsPath, value, { mode: 0o600 }))
-            .catch(() => undefined);
-    };
     const reads = new Map();
     const delivered = new WeakMap();
     let restart = false;
@@ -114,7 +109,6 @@ export async function startWorker(options) {
         }
     };
     const sendQuestions = async (root, targets, id) => {
-        remember(root);
         const read = (reads.get(root) ?? 0) + 1;
         reads.set(root, read);
         let value;
@@ -138,19 +132,24 @@ export async function startWorker(options) {
         }
     };
     const busy = () => [...runs.values()].some((run) => run.progress.status === 'running' || run.progress.status === 'stopping');
-    const close = async () => {
-        for (const socket of sockets.clients) {
-            socket.terminate();
-        }
-        await new Promise((resolve) => sockets.close(() => resolve()));
-        await new Promise((resolve) => server.close(() => resolve()));
-        await rm(options.socketPath, { force: true });
-        await saving;
-        await rm(options.lockPath, { force: true });
-        finish();
+    let closing;
+    let idleTimer;
+    const close = () => {
+        closing ??= (async () => {
+            clearTimeout(idleTimer);
+            for (const socket of sockets.clients) {
+                socket.terminate();
+            }
+            await new Promise((resolve) => sockets.close(() => resolve()));
+            await new Promise((resolve) => server.close(() => resolve()));
+            await rm(options.socketPath, { force: true });
+            await rm(options.lockPath, { force: true });
+            finish();
+        })();
+        return closing;
     };
-    const restartWhenIdle = () => {
-        if (restart && !busy()) {
+    const closeWhenIdle = () => {
+        if ((restart || sockets.clients.size === 0) && !busy()) {
             void close();
         }
     };
@@ -166,7 +165,6 @@ export async function startWorker(options) {
         }
         const run = { progress: startProgress(), stop: false };
         runs.set(root, run);
-        remember(root);
         const interrupts = {
             afterIssue: () => run.stop,
             afterRound: () => run.stop,
@@ -205,7 +203,7 @@ export async function startWorker(options) {
                     : 'failed';
             update({ ...run.progress, status });
         }, (error) => update({ ...run.progress, status: 'failed', error: message(error) }))
-            .finally(restartWhenIdle);
+            .finally(closeWhenIdle);
     };
     const stop = (root) => {
         const run = runs.get(root);
@@ -220,9 +218,8 @@ export async function startWorker(options) {
     sockets.on('connection', (socket) => {
         send(socket, { type: 'hello', version: options.version });
         send(socket, snapshot());
-        for (const root of roots) {
-            void sendQuestions(root, () => [socket]);
-        }
+        void sendQuestions(options.root, () => [socket]);
+        socket.on('close', closeWhenIdle);
         socket.on('message', (data) => {
             let value;
             try {
@@ -234,7 +231,7 @@ export async function startWorker(options) {
             if (value.type === 'hello') {
                 if (isNewer(value.version, options.version)) {
                     restart = true;
-                    restartWhenIdle();
+                    closeWhenIdle();
                 }
             }
             else if (value.type === 'start') {
@@ -261,11 +258,13 @@ export async function startWorker(options) {
             process.umask(umask);
         }
     });
+    idleTimer = setTimeout(closeWhenIdle, options.idleMs ?? IDLE_MS);
     return { closed, close };
 }
-export async function serve() {
-    const paths = socketPaths();
+export async function serve(root) {
+    const paths = workerPaths(root);
     const worker = await startWorker({
+        root,
         ...paths,
         version: await packageVersion(),
         runRepo,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "next-issue",
-  "version": "2.18.0",
+  "version": "2.20.0",
   "description": "Autonomous issue harness driven by the pi SDK",
   "keywords": [
     "github",