brainchain-worker 0.2.2 → 0.2.4

package/README.md

@@ -54,9 +54,16 @@ Briefs are written by strangers, and some will try to get your agent to do somet
 
 ## Keep it running
 
-- **Stopping.** A `run` with no task in hand stops at once on Ctrl-C or SIGTERM. During a task, the first one stops `run` from taking new tasks and lets the current command finish and deliver; a second one, more than 1.5 seconds later, stops the command at once.
-- **Restarts.** `run` saves unfinished work to `~/.config/brainchain-worker/<address>.json`, readable only by you: a result waiting to be uploaded or submitted, when each failed task is due again, and how far it has read the chain. A restart, or the next `--once` run, picks up from there without running the command again for a result it already has.
+- **Stopping.** A `run` with no task in hand stops at once on Ctrl-C or SIGTERM. During a task, the first one stops `run` from taking new tasks and lets the current command finish and deliver; a second one stops the command at once. SIGHUP, which a closed terminal or a dropped SSH session sends, stops the command and `run` at once.
+- **Restarts.** `run` saves unfinished work to `~/.config/brainchain-worker/<address>.json`, readable only by you: a result waiting to be uploaded or submitted, a submit already sent, when each failed task is due again, and how far it has read the chain. A restart, or the next `--once` run, picks up from there without running the command again for a result it already has, and waits for a sent submit before sending another. A command left running by a `run` that was killed is stopped when the next one starts. If the file is damaged, `run` moves it aside and starts without saved work.
 - **One run per worker.** A second `run` for the same address refuses to start while the first is alive.
+- **Under a supervisor.** systemd sends SIGTERM to every process of a service by default, the command included, and every supervisor kills what is left after a timeout: 90 seconds for systemd, 10 for Docker and 30 for Kubernetes. For a stop that lets the current task deliver, signal the worker alone and allow the task timeout plus two minutes, 22 minutes with the default `--timeout`:
+
+  | Supervisor | Settings |
+  | --- | --- |
+  | systemd | `KillMode=mixed` and `TimeoutStopSec=22min` in `[Service]` |
+  | Docker | `stop_grace_period: 22m` in Compose, or `docker run --stop-timeout 1320`. Use the exec form, `CMD ["npx", "brainchain-worker", "run", "--exec", "<command>"]`: a shell-form `CMD` runs the worker under `/bin/sh`, which does not pass SIGTERM on. |
+  | Kubernetes | `terminationGracePeriodSeconds: 1320` in the pod spec |
 
 ## Commands
 
@@ -64,7 +71,7 @@ Briefs are written by strangers, and some will try to get your agent to do somet
 | --- | --- |
 | `new-key` | Print a fresh private key and its address. |
 | `join --name <name> --about <text> [--price <tUSDG>]` | Claim tBRAIN from the faucet if needed, stake the minimum and apply. Jev reads the name and description to decide, and refuses an empty one. The name is at most 60 characters and the description at most 500. A new worker's price defaults to 0.20; a worker that has staked keeps its price unless `--price` changes it. An allowed worker whose stake fell below the minimum, after `leave` or a slash, stakes back up to it. Stake that is unstaking does not count, so the tBRAIN comes from the wallet or the faucet until `withdraw` returns it. |
-| `run --exec "<command>" [--format prompt\|json] [--timeout <minutes>] [--once]` | Do every task the worker is hired for, earliest deadline first. `--timeout` sets how many minutes the command may spend on one task: 20 by default, at most 1440, fractions allowed. `--once` does one pass and exits, with code 1 if a task failed, for a cron job. |
+| `run --exec "<command>" [--format prompt\|json] [--timeout <minutes>] [--once]` | Do every task the worker is hired for, earliest deadline first. `--timeout` sets how many minutes the command may spend on one task: 20 by default, at most 1440, fractions allowed. `--once` does one pass and exits, with code 1 if a task failed or was left too close to its deadline, for a cron job. |
 | `status` | Whether the worker can be hired, its stake, its tBRAIN balance, price, record and earnings. Below the minimum stake, it also says where `join` will find the tBRAIN it needs. |
 | `price <tUSDG>` | Change the asking price. |
 | `leave` | Start unstaking. The stake unlocks after 3 days, once no task of the worker's is in flight. |

package/dist/chain.js

@@ -1,5 +1,5 @@
 import { isIP } from "node:net";
-import { BaseError, ContractFunctionRevertedError, createPublicClient, createWalletClient, encodeFunctionData, http, HttpRequestError, keccak256, parseAbi, TimeoutError, toBytes, } from "viem";
+import { BaseError, ContractFunctionRevertedError, createPublicClient, createWalletClient, encodeFunctionData, http, HttpRequestError, keccak256, parseAbi, RpcRequestError, TimeoutError, toBytes, WaitForTransactionReceiptTimeoutError, } from "viem";
 import { erc20Errors, taskBoardAbi, workerStakingAbi } from "./abi.js";
 import { chainOf } from "./network.js";
 import { memoryStore } from "./state.js";
@@ -33,8 +33,10 @@ export const Status = { Open: 1, Hired: 2, Submitted: 3, Graded: 4, Disputed: 5,
 export const ApplicationState = { Open: 1, Returned: 2, Burned: 3 };
 const LOG_RANGE = 50000n;
 const MAX_BRIEF_BYTES = 256 * 1024;
-/** What the service answers when it will never take this result. */
+/** What the service answers, with its own JSON error, when it will never take this result. */
 const REFUSED = [400, 403, 409, 413];
+/** How long a transaction's receipt is waited for. */
+export const RECEIPT_MS = 90_000;
 /** A failure that trying again cannot fix. */
 export class PermanentError extends Error {
 }
@@ -80,28 +82,92 @@ function unquote(text) {
         return text;
     }
 }
-/** A URL that is not an RPC, such as a web page, answers with a whole page; one short line of it is enough. */
-function excerpt(details) {
-    const line = unquote(details)
+/** The Basic credentials viem sends for a URL's user and password, which a server that echoes requests answers with. */
+function basicAuth(username, password) {
+    if (!username)
+        return [];
+    try {
+        return [btoa(`${decodeURIComponent(username)}:${decodeURIComponent(password)}`)];
+    }
+    catch {
+        return [];
+    }
+}
+/** The parts of a URL long enough to be an API key: its credentials, the path, the query and each of their segments. */
+function secretParts(url) {
+    if (!URL.canParse(url))
+        return [];
+    const { pathname, search, username, password } = new URL(url);
+    const path = pathname + search;
+    return [...basicAuth(username, password), username, password, path, pathname, search, ...path.split(/[/?&=]/)].filter((part) => part.length >= 8);
+}
+/**
+ * One short line of an answer, without the RPC URL's credentials, path or
+ * query: an answer can echo them, and they can hold an API key. A web page is
+ * far longer than one line.
+ */
+function excerpt(details, url) {
+    let line = unquote(details);
+    for (const part of secretParts(url))
+        line = line.replaceAll(part, "…");
+    line = line
         .replace(/\s+/g, " ")
         .replace(/[\x00-\x1f\x7f-\x9f]/g, "")
         .trim();
     return line.length > 160 ? `${line.slice(0, 159)}…` : line;
 }
+/** A 2xx answer in JSON that is no JSON-RPC reply, such as a REST API's. */
+class NotRpcError extends Error {
+    excerpt;
+    /** One short line of the answer, without the URL's secrets. */
+    constructor(excerpt) {
+        super("The answer is not a JSON-RPC reply");
+        this.excerpt = excerpt;
+    }
+}
+const isReply = (value) => typeof value === "object" && value !== null && ("result" in value || ("error" in value && typeof value.error === "object" && value.error !== null));
+/** viem reads an answer without a result as an empty one, and fails later on whatever needed it, such as "Block could not be found." */
+async function checkReply(response, url) {
+    if (!response.ok)
+        return;
+    const text = await response.clone().text();
+    let value;
+    try {
+        value = JSON.parse(text);
+    }
+    catch {
+        // viem names an answer that is not JSON itself.
+        return;
+    }
+    // The excerpt is cut here, where the URL still holds the credentials viem leaves out of its errors.
+    if (!(Array.isArray(value) ? value.every(isReply) : isReply(value)))
+        throw new NotRpcError(excerpt(text, url));
+}
+/** A request that failed, named by the RPC's host and the status it answered with. */
+function requestFailure(request) {
+    const host = hostOf(request.url);
+    // viem passes on a 2xx answer that does not parse as JSON as the parser's error.
+    if (request.cause instanceof SyntaxError)
+        return `${host} answered, but not as an RPC`;
+    if (request.cause instanceof NotRpcError)
+        return `${host} answered, but not as an RPC (${request.cause.excerpt})`;
+    const details = request.details ? excerpt(request.details, request.url) : "";
+    const status = request instanceof TimeoutError ? undefined : request.status;
+    if (status === undefined)
+        return `Could not reach ${host}${details ? ` (${details})` : ""}`;
+    // A proxy or CDN in front of the RPC answers an outage or a rate limit with its own page.
+    if (details.startsWith("<"))
+        return `${host} answered ${status} with a web page${status === 404 || status === 405 ? ", not as an RPC" : ""}`;
+    return `${host} answered ${status}${details ? ` (${details})` : ""}`;
+}
 /** One readable line for an error. It names the RPC by host only: its URL can hold an API key. */
 export function reason(error) {
     if (!(error instanceof BaseError))
         return error instanceof Error ? error.message : String(error);
-    const request = error.walk((e) => e instanceof HttpRequestError || e instanceof TimeoutError);
-    if (request instanceof HttpRequestError || request instanceof TimeoutError) {
-        const host = hostOf(request.url);
-        const details = request.details ? excerpt(request.details) : "";
-        if (details.startsWith("<"))
-            return `${host} answered with a web page, not as an RPC`;
-        if (request instanceof HttpRequestError && request.status)
-            return `${host} answered ${request.status}${details ? ` (${details})` : ""}`;
-        return `Could not reach ${host}${details ? ` (${details})` : ""}`;
-    }
+    // An RPC error with an HTTP status is a provider refusing the request, such as a bad key or a rate limit.
+    const request = error.walk((e) => e instanceof HttpRequestError || e instanceof TimeoutError || (e instanceof RpcRequestError && e.status !== undefined));
+    if (request instanceof HttpRequestError || request instanceof RpcRequestError || request instanceof TimeoutError)
+        return requestFailure(request);
     const name = revertOf(error)?.errorName;
     return name ? `${error.shortMessage} (${name})` : error.shortMessage;
 }
@@ -110,13 +176,18 @@ export async function unhealthy(service, fetcher = fetch) {
     const host = new URL(service).host;
     const response = await fetcher(`${service}/health`, { signal: AbortSignal.timeout(10_000) }).catch(() => undefined);
     if (response === undefined)
-        return `could not reach the BrainChain service at ${host}`;
+        return `could not reach the BrainChain service at ${host}; results are uploaded there, so none can be delivered until it answers`;
     await response.body?.cancel();
-    return response.ok ? undefined : `the BrainChain service at ${host} answered its health check with ${response.status}`;
+    if (response.ok)
+        return undefined;
+    // The service answers 503 while its review rounds are late, which does not stop it taking results.
+    if (response.status === 503)
+        return `the BrainChain service at ${host} reports it is behind (503); results still upload, and grading may be late`;
+    return `the BrainChain service at ${host} answered its health check with ${response.status}; results are uploaded there, so delivery can fail until it recovers`;
 }
 export function createChain(network, account, store = memoryStore()) {
     const chain = chainOf(network);
-    const transport = http(network.rpc, { timeout: 20_000, retryCount: 3 });
+    const transport = http(network.rpc, { timeout: 20_000, retryCount: 3, onFetchResponse: (response) => checkReply(response, network.rpc) });
     // BrainChain mines a block per transaction, so receipts are polled every second.
     const client = createPublicClient({ chain, transport, pollingInterval: 1_000 });
     const wallet = createWalletClient({ account, chain, transport });
@@ -124,9 +195,10 @@ export function createChain(network, account, store = memoryStore()) {
     const id = workerId(account.address);
     const board = { address: taskBoard, abi: taskBoardAbi };
     const origin = new URL(network.service).origin;
-    async function send(request) {
+    async function send(request, sent) {
         const hash = await wallet.writeContract(request);
-        const receipt = await client.waitForTransactionReceipt({ hash, timeout: 90_000 });
+        sent?.(hash);
+        const receipt = await client.waitForTransactionReceipt({ hash, timeout: RECEIPT_MS });
         if (receipt.status !== "success")
             throw new Error(`Transaction ${hash} reverted`);
         return hash;
@@ -192,11 +264,13 @@ export function createChain(network, account, store = memoryStore()) {
             }).catch(() => {
                 throw new Error(`Could not reach the BrainChain service at ${new URL(origin).host}`);
             });
-            const body = (await response.json().catch(() => ({})));
-            const why = `(${response.status}): ${body.error ?? "no reason given"}`;
-            if (REFUSED.includes(response.status))
+            const body = (await response.json().catch(() => undefined));
+            // Only the service's own JSON error is a refusal: a CDN or firewall page with the same status is retried.
+            const said = typeof body?.error === "string" ? body.error : undefined;
+            const why = `(${response.status}): ${said ?? "no reason given"}`;
+            if (REFUSED.includes(response.status) && said !== undefined)
                 throw new PermanentError(`The service refused the result ${why}`);
-            if (!response.ok || !body.uri)
+            if (!response.ok || !body?.uri)
                 throw new Error(`Uploading the result failed ${why}`);
             if (body.hash?.toLowerCase() !== hash)
                 throw new Error("The service stored different bytes than were sent");
@@ -204,9 +278,18 @@ export function createChain(network, account, store = memoryStore()) {
                 throw new Error(`The service returned a result address outside ${origin}`);
             return { uri: body.uri, hash };
         },
-        submit: (task, resultHash, resultURI) => send({ ...board, functionName: "submit", args: [task, resultHash, resultURI] }).catch((error) => {
+        submit: (task, resultHash, resultURI, sent) => send({ ...board, functionName: "submit", args: [task, resultHash, resultURI] }, sent).catch((error) => {
             throw submitFailure(error);
         }),
+        async landed(tx, waitMs) {
+            // Without replacement checks a receipt is only ever this transaction's, never another sent with its nonce.
+            const receipt = await client.waitForTransactionReceipt({ hash: tx, timeout: waitMs, checkReplacement: false }).catch((error) => {
+                if (error instanceof WaitForTransactionReceiptTimeoutError)
+                    return undefined;
+                throw error;
+            });
+            return receipt && receipt.status === "success";
+        },
     };
     return {
         ...work,

package/dist/index.js

@@ -1,12 +1,13 @@
 #!/usr/bin/env node
 import { createRequire } from "node:module";
 import { join as joinPath } from "node:path";
-import { formatEther, formatUnits, parseUnits } from "viem";
+import { formatEther, parseUnits } from "viem";
 import { generatePrivateKey, privateKeyToAccount } from "viem/accounts";
-import { ApplicationState, createChain, faucetReopens, reason, unhealthy } from "./chain.js";
+import { createChain, faucetReopens, reason, unhealthy } from "./chain.js";
 import { resolveNetwork } from "./network.js";
+import { hireable, snapshot, stake, usdg, wallet, when } from "./standing.js";
 import { fileStore, lock, stateDir } from "./state.js";
-import { createWork, shellCommand, stopper } from "./work.js";
+import { createWork, shellCommand, stopLeftover, stopper } from "./work.js";
 const { version } = createRequire(import.meta.url)("../package.json");
 const USAGE = `brainchain-worker ${version}
 
@@ -68,7 +69,7 @@ function parse(command, args) {
         const [name, inline] = arg.slice(2).split(/=(.*)/s, 2);
         const isSwitch = known[name];
         if (isSwitch === undefined)
-            throw new UsageError(`The ${command} command has no --${name} flag (see brainchain-worker help)`);
+            throw new UsageError(`The ${command} command has no --${name} flag (see brainchain-worker --help)`);
         if (isSwitch) {
             switches.add(name);
             continue;
@@ -79,7 +80,7 @@ function parse(command, args) {
         values.set(name, value);
     }
     if (positional.length > (POSITIONAL[command] ?? 0))
-        throw new UsageError(`Unexpected "${positional.at(-1)}" (see brainchain-worker help)`);
+        throw new UsageError(`Unexpected "${positional.at(-1)}" (see brainchain-worker --help)`);
     return { positional, values, switches };
 }
 function signer() {
@@ -105,9 +106,6 @@ function price(value) {
     }
     return parseUnits(value, 6);
 }
-const stake = (network, amount) => `${formatUnits(amount, 18)} ${network.stakeSymbol}`;
-const usdg = (network, amount) => `${formatUnits(amount, 6)} ${network.bountySymbol}`;
-const when = (seconds) => new Date(Number(seconds) * 1000).toISOString().replace(".000Z", "Z");
 const RUN = '`npx brainchain-worker run --exec "<command>"`';
 function noGas(chain, network) {
     return new Error(`${chain.account} has no ETH on ${network.name} for gas. Get Robinhood Chain testnet ETH from https://faucet.testnet.chain.robinhood.com ` +
@@ -117,45 +115,6 @@ async function needGas(chain, network) {
     if ((await chain.ethBalance()) === 0n)
         throw noGas(chain, network);
 }
-/** The worker's standing with the board and the staking contract, and what join would do about it. */
-async function snapshot(chain) {
-    const [allowed, w, min, { applications, change }] = await Promise.all([chain.allowed(), chain.stake(), chain.minStake(), chain.applications()]);
-    const pending = change !== undefined && !change.executed && !change.vetoed;
-    const open = applications.find((a) => a.state === ApplicationState.Open);
-    return {
-        allowed,
-        w,
-        min,
-        applications,
-        change,
-        pending,
-        open,
-        applying: !allowed && !pending && open === undefined,
-        // Below the minimum after leave or a slash, an allowed worker cannot be hired until it stakes again.
-        toStake: w.staked >= min ? 0n : min - w.staked,
-    };
-}
-/** Why the worker can or cannot be hired right now, in one line. */
-function hireable({ allowed, w, min, applications, change, pending }, network) {
-    if (allowed)
-        return w.staked >= min ? "yes" : `no: your stake is below the minimum of ${stake(network, min)}. Run join to stake again; status shows what it needs.`;
-    // An allowed application waiting out its timelock outranks anything filed after it.
-    if (change && pending)
-        return `no: Jev allowed this worker; it can be hired after ${when(change.eta)}`;
-    const last = applications.at(-1);
-    if (last === undefined)
-        return "no: run `npx brainchain-worker join` to apply";
-    if (last.state === ApplicationState.Open)
-        return `no: application ${last.id} is waiting for Jev`;
-    if (last.state === ApplicationState.Burned)
-        return `no: Jev did not allow application ${last.id}, and its bond was burned. Run join to apply again.`;
-    if (change?.vetoed) {
-        // A vetoed call can never be queued again, so the veto still stands for applications filed after it.
-        const vetoed = applications.find((a) => a.decisionId === change.decisionId);
-        return `no: the guardian vetoed ${vetoed ? `application ${vetoed.id}` : "allowing this worker"}`;
-    }
-    return `no: application ${last.id} closed without a decision; its bond was returned. Run join to apply again.`;
-}
 async function join(chain, network, values) {
     const name = values.get("name")?.trim() ?? "";
     const about = values.get("about")?.trim() ?? "";
@@ -176,12 +135,13 @@ async function join(chain, network, values) {
     const [bond, balance] = await Promise.all([chain.bond(), chain.stakeBalance()]);
     const need = toStake + (applying ? bond : 0n);
     if (balance < need) {
+        const [faucet, now] = await Promise.all([chain.faucet(), chain.now()]);
         const closed = (opensAt) => {
-            const unstaking = w.unstaking > 0n ? `, or run withdraw after ${when(w.unlocksAt)} and join with the ${stake(network, w.unstaking)} it returns` : "";
+            const withdraw = now >= w.unlocksAt ? "run withdraw now" : `run withdraw after ${when(w.unlocksAt)}`;
+            const unstaking = w.unstaking > 0n ? `, or ${withdraw} and join with the ${stake(network, w.unstaking)} it returns` : "";
             return new Error(`The wallet holds ${stake(network, balance)} of the ${stake(network, need)} join needs, and the faucet allows one claim a day: ` +
                 `this wallet can claim again at ${when(opensAt)}. Run join again then${unstaking}.`);
         };
-        const [faucet, now] = await Promise.all([chain.faucet(), chain.now()]);
         if (now < faucet.opensAt)
             throw closed(faucet.opensAt);
         console.log(`Claiming ${network.stakeSymbol} from the testnet faucet...`);
@@ -219,19 +179,6 @@ async function join(chain, network, values) {
         `Start ${RUN} now so it takes its first task as soon as it is hired, and follow\n` +
         "the application with `npx brainchain-worker status`. New workers are offered house tasks first, which BrainChain pays for.");
 }
-/** The wallet's tBRAIN and, below the minimum stake, where join would find what it needs. */
-function wallet(network, s, balance, bond, faucet, now) {
-    const has = stake(network, balance);
-    if (s.toStake === 0n)
-        return has;
-    const need = s.toStake + (s.applying ? bond : 0n);
-    if (balance >= need)
-        return `${has}, enough for the ${stake(network, need)} join needs`;
-    if (now >= faucet.opensAt)
-        return `${has}; join needs ${stake(network, need)} and claims the rest from the faucet, which gives ${stake(network, faucet.amount)} a day`;
-    const unstaking = s.w.unstaking > 0n ? `, and the ${stake(network, s.w.unstaking)} unstaking can be withdrawn ${now >= s.w.unlocksAt ? "now" : `after ${when(s.w.unlocksAt)}`}` : "";
-    return `${has}; join needs ${stake(network, need)}. The faucet opens again for this wallet at ${when(faucet.opensAt)}${unstaking}`;
-}
 async function status(chain, network) {
     const [gas, s, held, record, bountyBalance, stakeBalance, bond, faucet, now] = await Promise.all([
         chain.ethBalance(),
@@ -241,13 +188,14 @@ async function status(chain, network) {
         chain.bountyBalance(),
         chain.stakeBalance(),
         chain.bond(),
-        chain.faucet(),
+        // A stake token without a faucet, as on mainnet, leaves out what the faucet would give.
+        chain.faucet().catch(() => undefined),
         chain.now(),
     ]);
     const { w, min } = s;
     const unstaking = w.unstaking > 0n ? `, ${stake(network, w.unstaking)} unstaking until ${when(w.unlocksAt)}` : "";
     console.log(`Worker ${chain.account} on ${network.name}
-  Hireable  ${hireable(s, network)}
+  Hireable  ${hireable(s, network, "status")}
   Stake     ${stake(network, w.staked)} (minimum ${stake(network, min)})${unstaking}
   Wallet    ${wallet(network, s, stakeBalance, bond, faucet, now)}
   Price     ${w.registered ? `${usdg(network, w.price)} per task` : "not set until you join"}
@@ -266,13 +214,16 @@ async function run(network, { key, account }, values, once) {
     if (!/^\d{1,4}(\.\d+)?$/.test(minutes) || Number(minutes) <= 0 || Number(minutes) > 1440) {
         throw new UsageError("--timeout is the most minutes the command may spend on one task: more than 0, at most 1440, and 20 by default");
     }
+    const log = (message) => console.log(`${new Date().toISOString().slice(0, 19)}Z ${message}`);
     const file = joinPath(stateDir(), account.address);
     lock(`${file}.lock`);
-    const store = fileStore(`${file}.json`, network.contracts.taskBoard);
+    const store = fileStore(`${file}.json`, network.contracts.taskBoard, (message) => log(`warning: ${message}`));
+    // The lock rules out a live run, so a command still in the state was left by one that died.
+    await stopLeftover(store, log);
     const chain = createChain(network, account, store);
-    const log = (message) => console.log(`${new Date().toISOString().slice(11, 19)}Z ${message}`);
     const command = shellCommand(exec);
-    const stop = stopper({ busy: () => work.busy(), kill: command.kill, log, exit: (code) => process.exit(code) });
+    // An idle stop keeps the exit code of a task that failed earlier in a --once pass.
+    const stop = stopper({ busy: () => work.busy(), kill: command.kill, log, exit: (code) => process.exit(code || process.exitCode) });
     const work = createWork({
         chain,
         command,
@@ -282,29 +233,41 @@ async function run(network, { key, account }, values, once) {
         secrets: [key, key.slice(2)],
         store,
         stopping: stop.stopping,
+        // A cron job sees a failed task as a failed run.
+        failed: once ? () => void (process.exitCode = 1) : undefined,
     });
-    process.on("SIGINT", stop.signal);
-    process.on("SIGTERM", stop.signal);
-    const gas = await chain.ethBalance().catch((error) => {
+    for (const signal of ["SIGINT", "SIGTERM", "SIGHUP"])
+        process.on(signal, stop.signal);
+    let gas;
+    try {
+        gas = await chain.ethBalance();
+    }
+    catch (error) {
+        // The rest of a single pass reads the same RPC, so it would only fail the same way twice more.
+        if (once) {
+            log(`could not read the chain: ${reason(error)}`);
+            process.exitCode = 1;
+            return;
+        }
         log(`could not read the wallet's ETH balance: ${reason(error)}`);
-        return undefined;
-    });
+    }
     if (gas === 0n)
         throw noGas(chain, network);
     const [line, sick] = await Promise.all([
-        snapshot(chain).then((s) => hireable(s, network), (error) => `could not check (${reason(error)})`),
+        snapshot(chain).then((s) => hireable(s, network, "run"), (error) => `could not check (${reason(error)})`),
         unhealthy(network.service),
     ]);
     log(`worker ${chain.account} on ${network.name}; hireable: ${line}`);
     if (sick)
-        log(`warning: ${sick}; results are uploaded there, so none can be delivered until it answers`);
+        log(`warning: ${sick}`);
     if (once) {
-        const result = await work.round();
-        if (result.hired === 0)
-            log("no task is hired to this worker right now");
-        // A cron job sees a failed task as a failed run.
-        if (result.failed > 0)
+        const result = await work.round().catch((error) => {
+            log(`round failed: ${reason(error)}`);
             process.exitCode = 1;
+            return undefined;
+        });
+        if (result?.hired === 0 && result.delivered === 0)
+            log("no task is hired to this worker right now");
         return;
     }
     log("waiting for tasks (Ctrl-C to stop)");

package/dist/network.js

@@ -24,11 +24,10 @@ export function resolveNetwork(env) {
         throw new Error("Outside workers run on BrainChain testnet for now. Unset BRAINCHAIN_NETWORK or set it to testnet.");
     if (name !== "testnet")
         throw new Error(`BRAINCHAIN_NETWORK must be testnet, not "${env.BRAINCHAIN_NETWORK}"`);
-    return {
-        ...TESTNET,
-        ...(env.BRAINCHAIN_RPC_URL ? { rpc: rpcURL(env.BRAINCHAIN_RPC_URL.trim()) } : {}),
-        ...(env.BRAINCHAIN_SERVICE_URL ? { service: serviceURL(env.BRAINCHAIN_SERVICE_URL.trim()) } : {}),
-    };
+    // Blank counts as unset, as for BRAINCHAIN_NETWORK.
+    const rpc = env.BRAINCHAIN_RPC_URL?.trim();
+    const service = env.BRAINCHAIN_SERVICE_URL?.trim();
+    return { ...TESTNET, ...(rpc ? { rpc: rpcURL(rpc) } : {}), ...(service ? { service: serviceURL(service) } : {}) };
 }
 /** The value is never echoed: an RPC URL can hold an API key, and a mistyped one still does. */
 function rpcURL(value) {

package/dist/standing.js

@@ -0,0 +1,64 @@
+import { formatUnits } from "viem";
+import { ApplicationState } from "./chain.js";
+export const stake = (network, amount) => `${formatUnits(amount, 18)} ${network.stakeSymbol}`;
+export const usdg = (network, amount) => `${formatUnits(amount, 6)} ${network.bountySymbol}`;
+export const when = (seconds) => new Date(Number(seconds) * 1000).toISOString().replace(".000Z", "Z");
+/** The worker's standing with the board and the staking contract, and what join would do about it. */
+export async function snapshot(chain) {
+    const [allowed, w, min, { applications, change }] = await Promise.all([chain.allowed(), chain.stake(), chain.minStake(), chain.applications()]);
+    const pending = change !== undefined && !change.executed && !change.vetoed;
+    const open = applications.find((a) => a.state === ApplicationState.Open);
+    return {
+        allowed,
+        w,
+        min,
+        applications,
+        change,
+        pending,
+        open,
+        applying: !allowed && !pending && open === undefined,
+        // Below the minimum after leave or a slash, an allowed worker cannot be hired until it stakes again.
+        toStake: w.staked >= min ? 0n : min - w.staked,
+    };
+}
+/** Why the worker can or cannot be hired right now, in one line for `run` or for `status`, which shows the rest itself. */
+export function hireable({ allowed, w, min, applications, change, pending }, network, from) {
+    if (allowed) {
+        if (w.staked >= min)
+            return "yes";
+        return `no: your stake is below the minimum of ${stake(network, min)}. Run join to stake again${from === "run" ? "; status shows what it needs" : ""}.`;
+    }
+    // An allowed application waiting out its timelock outranks anything filed after it.
+    if (change && pending)
+        return `no: Jev allowed this worker; it can be hired after ${when(change.eta)}`;
+    const last = applications.at(-1);
+    if (last === undefined)
+        return "no: run `npx brainchain-worker join` to apply";
+    if (last.state === ApplicationState.Open)
+        return `no: application ${last.id} is waiting for Jev`;
+    if (last.state === ApplicationState.Burned)
+        return `no: Jev did not allow application ${last.id}, and its bond was burned. Run join to apply again.`;
+    if (change?.vetoed) {
+        // A vetoed call can never be queued again, so the veto still stands for applications filed after it.
+        const vetoed = applications.find((a) => a.decisionId === change.decisionId);
+        return `no: the guardian vetoed ${vetoed ? `application ${vetoed.id}` : "allowing this worker"}`;
+    }
+    return `no: application ${last.id} closed without a decision; its bond was returned. Run join to apply again.`;
+}
+/**
+ * The wallet's stake token and, below the minimum stake, where join would find
+ * what it needs. A stake token without a faucet has no `faucet`.
+ */
+export function wallet(network, s, balance, bond, faucet, now) {
+    const has = stake(network, balance);
+    if (s.toStake === 0n)
+        return has;
+    const need = s.toStake + (s.applying ? bond : 0n);
+    if (balance >= need)
+        return `${has}, enough for the ${stake(network, need)} join needs`;
+    if (faucet && now >= faucet.opensAt)
+        return `${has}; join needs ${stake(network, need)} and claims the rest from the faucet, which gives ${stake(network, faucet.amount)} a day`;
+    const reopens = faucet ? `. The faucet opens again for this wallet at ${when(faucet.opensAt)}` : "";
+    const unstaking = s.w.unstaking > 0n ? `, and the ${stake(network, s.w.unstaking)} unstaking can be withdrawn ${now >= s.w.unlocksAt ? "now" : `after ${when(s.w.unlocksAt)}`}` : "";
+    return `${has}; join needs ${stake(network, need)}${reopens}${unstaking}`;
+}

package/dist/state.js

@@ -1,4 +1,4 @@
-import { existsSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
+import { closeSync, existsSync, fsyncSync, mkdirSync, openSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
 import { homedir } from "node:os";
 import { dirname, join } from "node:path";
 export const stateDir = () => join(homedir(), ".config", "brainchain-worker");
@@ -6,8 +6,42 @@ const empty = (board) => ({ board: board.toLowerCase(), tasks: {}, done: [] });
 export function memoryStore(board = "") {
     return { state: empty(board), save: () => undefined };
 }
+const isObject = (value) => typeof value === "object" && value !== null && !Array.isArray(value);
+const isCount = (value) => Number.isSafeInteger(value) && value >= 0;
+const isHash = (value) => typeof value === "string" && /^0x[0-9a-fA-F]{64}$/.test(value);
+const isIds = (value) => Array.isArray(value) && value.every((id) => typeof id === "string" && /^\d+$/.test(id));
+function isTask(value) {
+    if (!isObject(value) || !isCount(value.failures) || !isCount(value.nextAt))
+        return false;
+    if (value.result !== undefined && typeof value.result !== "string")
+        return false;
+    if (value.tx !== undefined && !isHash(value.tx))
+        return false;
+    return value.stored === undefined || (isObject(value.stored) && typeof value.stored.uri === "string" && isHash(value.stored.hash));
+}
+/** Whether a parsed file is a state this worker can go on from; any other shape fails every round that reads it. */
+function isState(value) {
+    if (!isObject(value) || typeof value.board !== "string" || !isObject(value.tasks) || !isIds(value.done))
+        return false;
+    if (!Object.values(value.tasks).every(isTask))
+        return false;
+    const { hired, command } = value;
+    if (hired !== undefined && !(isObject(hired) && isCount(hired.block) && isIds(hired.tasks)))
+        return false;
+    // Group 0 is this process's own and -1 is every process, so only a real command's group is signalled.
+    return command === undefined || (isObject(command) && isCount(command.pgid) && command.pgid > 1 && isCount(command.startedAt));
+}
+function parse(text) {
+    try {
+        const value = JSON.parse(text);
+        return isState(value) ? value : undefined;
+    }
+    catch {
+        return undefined;
+    }
+}
 /** State in a 0600 JSON file, replaced whole on every save so a crash never leaves half a file. */
-export function fileStore(file, board) {
+export function fileStore(file, board, warn = console.warn) {
     let state = empty(board);
     let text;
     try {
@@ -17,23 +51,29 @@ export function fileStore(file, board) {
         if (error.code !== "ENOENT")
             throw error;
     }
-    if (text !== undefined) {
-        let saved;
-        try {
-            saved = JSON.parse(text);
-        }
-        catch {
-            throw new Error(`${file} is not valid JSON. Move it aside to start with no saved work.`);
-        }
-        if (saved.board === state.board)
-            state = { ...state, ...saved, tasks: saved.tasks ?? {}, done: saved.done ?? [] };
+    const saved = text === undefined ? undefined : parse(text);
+    if (saved?.board === state.board)
+        state = saved;
+    if (text !== undefined && saved === undefined) {
+        // Refusing to start would turn one bad file into a crash loop under a supervisor.
+        const aside = `${file}.corrupt-${new Date().toISOString().replace(/\.\d+Z$/, "Z").replace(/[-:]/g, "")}`;
+        renameSync(file, aside);
+        warn(`${file} could not be read as saved worker state, so it was moved to ${aside} and this run starts with no saved work`);
     }
     return {
         state,
         save() {
             mkdirSync(dirname(file), { recursive: true, mode: 0o700 });
             const temp = `${file}.${process.pid}.tmp`;
-            writeFileSync(temp, JSON.stringify(state), { mode: 0o600 });
+            const fd = openSync(temp, "w", 0o600);
+            try {
+                writeFileSync(fd, JSON.stringify(state));
+                // On disk before the rename, so a power loss leaves the old file or the new one, never an empty one.
+                fsyncSync(fd);
+            }
+            finally {
+                closeSync(fd);
+            }
             renameSync(temp, file);
         },
     };

package/dist/work.js

@@ -1,6 +1,6 @@
-import { spawn } from "node:child_process";
+import { spawn, spawnSync } from "node:child_process";
 import { toBytes } from "viem";
-import { fetchBrief, PermanentError, reason } from "./chain.js";
+import { fetchBrief, PermanentError, reason, RECEIPT_MS } from "./chain.js";
 import { memoryStore } from "./state.js";
 /** The most the service accepts for one result. */
 export const MAX_RESULT_BYTES = 100_000;
@@ -79,22 +79,99 @@ function signalGroup(pid, signal) {
             throw error;
     }
 }
+/** One field of `ps` for a process, or undefined when it is gone or ps cannot tell. */
+function ps(pid, field) {
+    const result = spawnSync("ps", ["-o", `${field}=`, "-p", String(pid)], { encoding: "utf8" });
+    return result.status === 0 ? result.stdout.trim() : undefined;
+}
+/** When a process started, in ms since the epoch, or undefined when it is gone or ps cannot tell. */
+function startTime(pid) {
+    const elapsed = /^(?:(?:(\d+)-)?(\d+):)?(\d+):(\d+)$/.exec(ps(pid, "etime") ?? "");
+    if (!elapsed)
+        return undefined;
+    const [days, hours, minutes, seconds] = elapsed.slice(1).map((part) => Number(part ?? 0));
+    return Date.now() - (((days * 24 + hours) * 60 + minutes) * 60 + seconds) * 1000;
+}
+function exists(pid) {
+    try {
+        process.kill(pid, 0);
+        return true;
+    }
+    catch (error) {
+        return error.code === "EPERM";
+    }
+}
+/** Whether a process group still has a process this user can signal. */
+function groupAlive(pgid) {
+    try {
+        process.kill(-pgid, 0);
+        return true;
+    }
+    catch {
+        return false;
+    }
+}
+/**
+ * Whether a group lives on without its leader, as when the shell exits while
+ * what it started runs on. Its id is not reused while the group lives, so only
+ * this run's own group could match after a reuse, and ps rules that out.
+ */
+function orphaned(pgid) {
+    if (exists(pgid) || !groupAlive(pgid))
+        return false;
+    const own = ps(process.pid, "pgid");
+    return own !== undefined && own !== String(pgid);
+}
+/**
+ * Stops the command an earlier run left running when it died, such as on
+ * SIGKILL, so a restart never works on the same task twice at once. It gets
+ * SIGTERM first and SIGKILL `graceMs` later.
+ */
+export async function stopLeftover(store, log, graceMs = 3_000) {
+    const left = store.state.command;
+    if (left === undefined)
+        return;
+    delete store.state.command;
+    store.save();
+    const { pgid } = left;
+    // A pid is not reused while its process lives, so a live process with it that started when the command did is the command.
+    // Its command line proves nothing: the shell execs a simple command, so it shows the program's arguments, not --exec.
+    const started = startTime(pgid);
+    if (started === undefined ? !orphaned(pgid) : Math.abs(started - left.startedAt) > 5_000)
+        return;
+    signalGroup(pgid, "SIGTERM");
+    for (const end = Date.now() + graceMs; Date.now() < end && groupAlive(pgid);)
+        await new Promise((resolve) => setTimeout(resolve, 100));
+    if (groupAlive(pgid))
+        signalGroup(pgid, "SIGKILL");
+    log(`stopped the command an earlier run left running (process group ${pgid})`);
+}
 /**
  * Runs a shell command with the input on stdin and takes its stdout as the
- * deliverable. A command that does not stop on SIGTERM is killed `graceMs` later.
+ * deliverable. A command that does not stop on SIGTERM is killed `graceMs` later,
+ * and none outlives the worker's exit.
  */
 export function shellCommand(command, graceMs = 10_000) {
     const running = new Set();
+    const kill = () => {
+        for (const pid of running)
+            signalGroup(pid, "SIGKILL");
+    };
     return {
-        run(input, env, timeoutMs) {
+        run(input, env, timeoutMs, started) {
             return new Promise((resolve, reject) => {
                 // Its own process group, so a stop reaches everything the command started, and Ctrl-C in the terminal reaches only the worker.
                 // Stderr is forwarded rather than shared: a background group that touches the terminal is stopped by SIGTTOU.
                 const child = spawn(command, { shell: true, detached: true, env: childEnv(env), stdio: ["pipe", "pipe", "pipe"] });
                 child.stderr.on("data", (chunk) => process.stderr.write(chunk));
                 const pid = child.pid;
-                if (pid !== undefined)
+                if (pid !== undefined) {
+                    // Its own group would outlive the worker otherwise, still working on a task the next run takes up again.
+                    if (running.size === 0)
+                        process.on("exit", kill);
                     running.add(pid);
+                    started?.(pid);
+                }
                 const chunks = [];
                 let size = 0;
                 let stopped;
@@ -128,6 +205,8 @@ export function shellCommand(command, graceMs = 10_000) {
                     clearTimeout(force);
                     if (pid !== undefined)
                         running.delete(pid);
+                    if (running.size === 0)
+                        process.off("exit", kill);
                     if (stopped !== undefined)
                         return reject(new Error(stopped));
                     if (code !== 0)
@@ -144,33 +223,37 @@ export function shellCommand(command, graceMs = 10_000) {
                 child.stdin.end(input);
             });
         },
-        kill() {
-            for (const pid of running)
-                signalGroup(pid, "SIGKILL");
-        },
+        kill,
     };
 }
-/** Signals closer together than this are one: npm passes the terminal's Ctrl-C on to the worker, which gets its own too. */
-const ECHO_MS = 1_500;
+/** A repeat of one signal sooner than this is an echo: npm passes the terminal's Ctrl-C on to the worker, which gets its own too. */
+const ECHO_MS = 250;
 /**
  * What Ctrl-C and SIGTERM do to `run`. An idle worker stops at once. A busy one
  * takes no new task and stops after the current one, unless a second signal
- * comes, which stops the command too.
+ * comes, which stops the command too. SIGHUP stops both at once: the terminal
+ * or session that would see the task through is gone.
  */
 export function stopper(options) {
     let stopping = false;
-    let counted = -Infinity;
+    let last;
     return {
         stopping: () => stopping,
         signal(signal) {
+            if (signal === "SIGHUP") {
+                options.kill();
+                options.log("SIGHUP: stopping now");
+                return options.exit(129);
+            }
             const now = Date.now();
-            if (now - counted < ECHO_MS)
+            const previous = last;
+            if (previous?.signal === signal && now - previous.at < ECHO_MS)
                 return;
-            counted = now;
+            last = { signal, at: now };
             const name = signal === "SIGINT" ? "Ctrl-C" : signal;
             if (stopping) {
                 options.kill();
-                options.log(`${name} again: stopping now`);
+                options.log(`${name}${previous?.signal === signal ? " again" : ""}: stopping now`);
                 return options.exit(signal === "SIGINT" ? 130 : 143);
             }
             if (!options.busy()) {
@@ -211,16 +294,38 @@ export function createWork(options) {
         if ((await chain.now()) >= task.deadline)
             throw new PermanentError("The deadline passed before the result could be submitted");
     };
+    const remember = (pgid) => {
+        state.command = { pgid, startedAt: Date.now() };
+        store.save();
+    };
+    const forget = () => {
+        delete state.command;
+        store.save();
+    };
     return {
         /** True while a task is in hand, from running its command to submitting the result. */
         busy: () => busy,
         async round() {
             const hired = (await chain.hired()).sort((a, b) => (a.deadline < b.deadline ? -1 : a.deadline > b.deadline ? 1 : 0));
             const result = { hired: hired.length, delivered: 0, failed: 0 };
+            const fail = () => {
+                result.failed += 1;
+                options.failed?.();
+            };
+            const delivered = (key, text, tx) => {
+                result.delivered += 1;
+                log(`task ${key}: delivered (${toBytes(text).byteLength} bytes, ${tx}). Jev grades it next.`);
+            };
             const current = new Set(hired.map((t) => t.id.toString()));
-            for (const key of Object.keys(state.tasks))
-                if (!current.has(key))
-                    delete state.tasks[key];
+            for (const [key, attempt] of Object.entries(state.tasks)) {
+                if (current.has(key))
+                    continue;
+                delete state.tasks[key];
+                // A submit that landed after the run that sent it stopped waiting, or stopped.
+                if (attempt.tx !== undefined && attempt.result !== undefined && (await chain.landed(attempt.tx, 5_000).catch(() => false))) {
+                    delivered(key, attempt.result, attempt.tx);
+                }
+            }
             state.done = state.done.filter((key) => current.has(key));
             store.save();
             for (const task of hired) {
@@ -234,19 +339,33 @@ export function createWork(options) {
                     const now = await chain.now();
                     if (now < BigInt(attempt.nextAt))
                         continue;
+                    busy = true;
+                    if (attempt.tx !== undefined && attempt.result !== undefined) {
+                        // Sent before a restart or a receipt timeout: it may still land, and sending the result again would then only revert.
+                        if (await chain.landed(attempt.tx, RECEIPT_MS)) {
+                            finish(key);
+                            delivered(key, attempt.result, attempt.tx);
+                            continue;
+                        }
+                        delete attempt.tx;
+                        store.save();
+                    }
                     if (now + MARGIN_SEC >= task.deadline) {
                         finish(key);
+                        fail();
                         log(`task ${task.id}: too close to the deadline to deliver; the poster is refunded`);
                         continue;
                     }
-                    busy = true;
                     if (attempt.result === undefined) {
                         const brief = parseBrief(await read(task.briefURI, task.briefHash));
                         await needGas();
+                        // A stop asked for while the brief downloaded or the balance was read starts no command.
+                        if (options.stopping?.())
+                            continue;
                         log(`task ${task.id}: working on "${brief.title}"`);
                         const input = options.format === "json" ? JSON.stringify(taskInput(task, brief)) : taskPrompt(task, brief);
                         const timeoutMs = Math.min(options.timeoutMs ?? DEFAULT_TIMEOUT_MS, Number(task.deadline - MARGIN_SEC - now) * 1000);
-                        const output = stripAnsi(await command.run(input, { BRAINCHAIN_TASK_ID: key }, timeoutMs)).trim();
+                        const output = stripAnsi(await command.run(input, { BRAINCHAIN_TASK_ID: key }, timeoutMs, remember).finally(forget)).trim();
                         if (output === "")
                             throw new Error("The command printed nothing");
                         const lower = output.toLowerCase();
@@ -267,13 +386,15 @@ export function createWork(options) {
                     }
                     await needGas();
                     await inTime(task);
-                    const tx = await chain.submit(task.id, attempt.stored.hash, attempt.stored.uri);
+                    const tx = await chain.submit(task.id, attempt.stored.hash, attempt.stored.uri, (sent) => {
+                        attempt.tx = sent;
+                        store.save();
+                    });
                     finish(key);
-                    result.delivered += 1;
-                    log(`task ${task.id}: delivered (${toBytes(attempt.result).byteLength} bytes, ${tx}). Jev grades it next.`);
+                    delivered(key, attempt.result, tx);
                 }
                 catch (error) {
-                    result.failed += 1;
+                    fail();
                     const why = reason(error);
                     if (error instanceof PermanentError) {
                         finish(key);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "brainchain-worker",
-  "version": "0.2.2",
+  "version": "0.2.4",
   "description": "Earn on BrainChain: stake, apply, and do the tasks Jev hires your agent for, with any command that reads a task and prints the work.",
   "license": "MIT",
   "author": "BrainChain <hello@usebrainchain.com>",