@unotest/core 0.25.0 → 0.26.1

package/CHANGELOG.md

@@ -1,5 +1,129 @@
 # Changelog
 
+## [0.26.1] - 2026-08-30
+
+### Patch Changes
+
+- `packBundle` accepts `declaredVariables` — the external NAMES a suite
+  declares, carried in the manifest so a box can tell a pusher which of
+  them it has no value for. Names only; the files that hold the values
+  are excluded from every bundle as before.
+
+- Updated dependencies [9424f6b]
+  - @unotest/protocol@0.26.1
+  - @unotest/dsl@0.26.1
+
+## [0.26.0] - 2026-08-30
+
+### Minor Changes
+
+- 6d1c612: `npx @unotest/web bundle push` — send your suite to a box, instead of
+  giving a box access to your repository.
+
+  ```sh
+  npx @unotest/web bundle push --box https://tests.example.com
+  ```
+
+  It packs `unotest/`, `unotest.config`, `package.json` and its lockfile
+  into one archive and uploads it. That is the whole channel: a box holds
+  no deploy key and no token for anybody's code and never pulls anything,
+  so nothing is on it that was not pushed to it. (There is no path-scoped
+  git credential anywhere — a key that can fetch `unotest/` can fetch the
+  product source next to it, which is why this direction is the only one
+  we offer.)
+
+  - **Uncommitted work travels.** The manifest records `dirty: true` and
+    the bundle shows up as _wip_, so pushing a fix you have not committed
+    yet is a normal loop, not a release step.
+  - **What would fail on the box is refused here.** A scenario that reads a
+    file outside `unotest/` (`MODULE_NOT_FOUND` on the box, hours later, on
+    a run nobody is watching), a missing lockfile (`npm ci` will not run
+    without one), no `@unotest/web` in the dependencies, a symlink. Each
+    problem names the file, the line and the fix.
+  - **Secrets do not travel.** `unotest/.env*` and `.secrets*` are excluded
+    even if committed: environment values belong to the environment and are
+    injected over the bundle when it runs.
+  - **In a git project, git decides what belongs.** The file list comes from
+    `git ls-files --cached --others --exclude-standard`, so whatever your
+    `.gitignore` keeps out — generated databases, recordings, `.runs` —
+    stays out. Content is read from the working tree, not the index.
+  - **The id is the content.** `bundleId` is a hash of the packed files, not
+    a commit sha (a dirty tree's sha is not unique), so re-pushing an
+    unchanged suite is an instant no-op and two people packing the same tree
+    get the same bundle.
+  - `--dry-run` packs and checks without uploading, `--out <file>` keeps the
+    archive (plain `tar.gz` — `tar -tzf` shows what is inside), `--json`
+    prints `{ bundleId, status }` for a CI job that runs it afterwards.
+  - `UNOTEST_BOX_URL` / `UNOTEST_BOX_TOKEN` supply the address and the
+    token, which a CI job usually takes from its secret store. Tokens are
+    issued per project on the box.
+
+- c43aa00: Runs of one project now take turns instead of colliding.
+
+  Until now nothing coordinated them: the viewer refused a second run with
+  HTTP 409, and that was the whole defence — a `npx @unotest/web e2e` in a
+  terminal, an agent's `run_test` and a scheduled suite would all start on
+  top of each other, driving the same browser and the same seeded database
+  at the same time. The 409 protected the UI, not the machine.
+
+  - **A filesystem queue**, in `unotest/.queue<target>[.<env>]/` beside the
+    runs root it belongs to. A run writes a ticket, waits until it is at
+    the head, takes a slot, runs, and gives the slot back. There is no
+    daemon: whoever holds the slot executes the run itself, so nothing new
+    has to be installed, started or kept alive. Only `open("wx")`, rename,
+    unlink and mtime — no `flock`, which lies on network and container
+    filesystems. A crashed producer is reaped by whoever comes next.
+  - **Every producer is in it**: the CLI (`e2e`, `collection`, `author`),
+    the viewer's Run button, and the MCP `run_test` (which spawns the CLI).
+    A collection is ONE run — its `workers` still run in parallel inside
+    it, because a child process carries its parent's lease and skips the
+    queue.
+  - **The viewer shows the queue**: the Active panel lists what is waiting,
+    including tickets a terminal or an agent wrote, with a button to take
+    any of them back out. `GET /api/queue`, `DELETE /api/queue/:ticket`,
+    and a `queue:changed` websocket message.
+  - **Config**: `queue.concurrency` (default 1) and `queue.enabled`
+    (default true) in `unotest.config`. Both the CLI and the viewer read
+    the same file, so they cannot disagree about how many slots exist.
+  - **Escape hatches**: `UNOTEST_NO_QUEUE=1` runs without queueing;
+    Ctrl-C while waiting gives up your place in line and runs nothing.
+  - **For boxes**: `UNOTEST_QUEUE_GLOBAL_DIR` + `UNOTEST_QUEUE_GLOBAL_SLOTS`
+    add a host-wide budget several environments share (a collection weighs
+    its worker count), and `UNOTEST_ARTIFACTS_ROOT` puts `.runs` / `.queue`
+    somewhere other than the directory the tests live in — so the history
+    outlives a throwaway copy of the tests.
+
+  ### Migration notes
+
+  **`POST /api/run` no longer answers 409.** Ordering a run while another
+  one is active used to fail with `ActiveRunConflictError` (HTTP 409); it
+  now succeeds and the run waits. The response changed shape with it:
+
+  ```diff
+  - { runId, kind, ref, pid, startedAt }
+  + { runId, kind, ref, status: "queued" | "running", ticket, acceptedAt }
+  ```
+
+  `runId` is still final and immediately usable — open the tab on it as
+  before — but the run may not have started yet, and `pid` is not known at
+  that point. `POST /api/runs/:runId/abort` covers both states: it
+  withdraws a waiting ticket or SIGTERMs a running child.
+
+  Anything that treated 409 as "busy, try later" should now simply order
+  the run. Anything that relied on "only one run can exist" should read
+  `GET /api/queue`. To keep the old immediate-start behaviour (without the
+  protection), set `queue.enabled: false`.
+
+### Patch Changes
+
+- Updated dependencies [3eede7c]
+- Updated dependencies [6d1c612]
+- Updated dependencies [dbfa36f]
+- Updated dependencies [c43aa00]
+- Updated dependencies [8ec4177]
+  - @unotest/protocol@0.26.0
+  - @unotest/dsl@0.26.0
+
 ## [0.25.0] - 2026-08-29
 
 ### Patch Changes

package/README.md

@@ -13,6 +13,13 @@ filesystem layer both `@unotest/web` and `@unotest/mobile` need:
   `process.env` always winning.
 - **`appendUniqueLines`** — idempotent `.gitignore` (or any line-based
   file) updater used by the runners' `init`.
+- **`FsRunQueue`** — the filesystem run queue (`unotest/.queue…`) that
+  serialises RUNS of one project + environment across every producer:
+  a CLI in a terminal, the viewer's Run button, an agent. Ticket +
+  slot-lease model, with the crash recovery that implies (mtime
+  liveness, rename-quarantine reaping). Uses only `open("wx")`, rename,
+  unlink and mtime — never `flock`, which lies on the network and
+  container filesystems this has to work on.
 
 Depends only on `@unotest/protocol`. Unlike protocol (pure data, no fs),
 this package deliberately owns filesystem side effects.

package/dist/index.d.ts

@@ -1,4 +1,5 @@
-import { RunArtifact, RunManifest, RunSources, RuntimeStateFile, RuntimeState, DbgCommand, PickerMode, DbgCommandRecord, DbgCommandProbe } from '@unotest/protocol';
+import { RunArtifact, RunManifest, RunSources, RuntimeStateFile, RuntimeState, DbgCommand, PickerMode, DbgCommandRecord, DbgCommandProbe, IRunQueue, EnqueueRunInput, RunQueueTicketHandle, QueueSnapshot, QueueTicket, RunQueueLease, BundleGitInfo, BundleManifest } from '@unotest/protocol';
+export { isRecord } from '@unotest/protocol';
 import { ExecutionEvent } from '@unotest/dsl/executor';
 
 interface IRunArtifactWriter {
@@ -259,4 +260,225 @@ declare class UnotestError extends Error {
     constructor(message: string, context?: Record<string, unknown>);
 }
 
-export { type ArtifactRedactor, type DebugCommandsWatcherDeps, type DebugControlTarget, type DebugWatcherLogger, type E2EFlags, type GitignoreUpdate, type IRunArtifactWriter, JsonlRunArtifactWriter, type JsonlRunArtifactWriterOptions, type RunHeartbeat, type RuntimeExecState, type RuntimeInspection, type RuntimeInspectionInput, type RuntimeStateExtra, type RuntimeStateWriter, type RuntimeStateWriterDeps, UnotestError, type WriteManifestInput, type WriteRunSourcesInput, appendUniqueLines, applyEnvLayers, buildRuntimeInspection, createRunArtifactWriter, createRuntimeStateWriter, currentLocation, extractCallStack, levenshteinDistance, parseE2EFlags, readDebuggerBreakpoints, readEnvFile, readEnvLayers, startDebugCommandsWatcher, startRunHeartbeat, suggestClosest, toProtocolRuntimeState, writeRunManifest, writeRunSources };
+interface GlobalPoolOptions {
+    /** Absolute directory shared by every environment on this host
+     *  (`UNOTEST_QUEUE_GLOBAL_DIR`; in compose, a shared volume). */
+    dir: string;
+    /** Total WEIGHT the host can run at once
+     *  (`UNOTEST_QUEUE_GLOBAL_SLOTS`). */
+    slots: number;
+}
+/** Read the optional host-wide weight pool from the environment. Present
+ *  on a box where several environments share one machine's CPU and RAM;
+ *  absent for an npm user, who then only has the per-environment queue.
+ *  The ONE parser for every producer — the CLI and the viewer counting
+ *  different pools over the same directory would defeat the pool. */
+declare function globalPoolFromEnv(env?: NodeJS.ProcessEnv): GlobalPoolOptions | null;
+interface FsRunQueueOptions {
+    /** Absolute queue root — `<artifacts root>/unotest/.queue[…]`. */
+    root: string;
+    /** Environment concurrency: how many RUNS may be in flight here. */
+    slots: number;
+    global?: GlobalPoolOptions;
+    staleAfterMs?: number;
+    touchIntervalMs?: number;
+    pollIntervalMs?: number;
+    /** Injected clock + entropy + timer, so the tests can drive staleness
+     *  and races deterministically instead of sleeping through them. */
+    now?: () => number;
+    random?: () => string;
+    sleep?: (ms: number) => Promise<void>;
+}
+declare class FsRunQueue implements IRunQueue {
+    private readonly root;
+    private readonly ticketsDir;
+    private readonly runningDir;
+    private readonly staleAfterMs;
+    private readonly touchIntervalMs;
+    private readonly pollIntervalMs;
+    private readonly now;
+    private readonly random;
+    private readonly sleep;
+    private readonly envPool;
+    private readonly globalPool;
+    constructor(opts: FsRunQueueOptions);
+    enqueue(input: EnqueueRunInput): Promise<RunQueueTicketHandle>;
+    snapshot(): Promise<QueueSnapshot>;
+    cancel(ticketName: string): Promise<boolean>;
+    /** Reap what died, then drop `running/` entries with no lease behind
+     *  them. Every producer calls this on its way past — a queue with any
+     *  traffic needs no janitor process. */
+    sweep(): Promise<void>;
+    /** Wait until `handle` owns a slot. Public only for the handle it
+     *  belongs to. */
+    acquire(handle: FsTicketHandle, signal?: AbortSignal): Promise<RunQueueLease>;
+    /** One claim attempt for the ticket sitting at `index` of the live
+     *  queue. Null means "not now" — the caller waits and retries.
+     *
+     *  Position rule: a ticket may attempt a claim when its index is below
+     *  the environment's slot count. With the default concurrency of 1
+     *  that is strict FIFO; above it, two eligible tickets may take a
+     *  freed slot out of order, which is the price of not serialising
+     *  every waiter behind the head of the queue.
+     *
+     *  A maintenance ticket is exclusive: it waits for the head position
+     *  and then takes EVERY slot, so nothing else runs while it mutates
+     *  the environment. While one is at the head, NOBODY else may claim —
+     *  without that drain rule a steady stream of later tickets would keep
+     *  grabbing each freed slot and the head's all-or-nothing claim would
+     *  starve forever. Detected from ticket names (the maintenance rank is
+     *  the first character), so waiters never read each other's files. */
+    private tryClaim;
+    /** Live waiting tickets, in service order. */
+    private waitingOrder;
+    private readEntries;
+    private clampWeight;
+    /** Heartbeat + teardown for a lease — kept here so `FsLease` stays a
+     *  handle and the filesystem knowledge stays in one class. */
+    releaseLease(handle: FsTicketHandle, envSlots: readonly string[], globalSlots: readonly string[] | null): Promise<void>;
+    releaseLeaseSync(handle: FsTicketHandle, envSlots: readonly string[], globalSlots: readonly string[] | null): void;
+    touchLease(handle: FsTicketHandle, envSlots: readonly string[], globalSlots: readonly string[] | null): Promise<void>;
+    get heartbeatIntervalMs(): number;
+}
+declare class FsTicketHandle implements RunQueueTicketHandle {
+    private readonly queue;
+    readonly name: string;
+    readonly ticket: QueueTicket;
+    constructor(queue: FsRunQueue, name: string, ticket: QueueTicket);
+    acquire(opts?: {
+        signal?: AbortSignal;
+    }): Promise<RunQueueLease>;
+    cancel(): Promise<void>;
+}
+/** Env every child of a lease holder inherits. The boundary the queue
+ *  draws is "top-level run": a collection's scenario subprocesses carry
+ *  this and skip the queue, so `workers` keeps meaning what it always
+ *  meant instead of deadlocking behind its own parent. */
+declare function leaseEnv(runId: string): Readonly<Record<string, string>>;
+
+/** The ticket disappeared while its owner was waiting for a slot: it was
+ *  cancelled (viewer `DELETE /api/queue/:ticket`) or reaped as stale
+ *  (the owner was suspended long enough to look dead). Callers treat
+ *  this as "this run will never start" — it is not a failure of the run,
+ *  and there is nothing to retry automatically. */
+declare class QueueTicketLostError extends UnotestError {
+    constructor(ticketName: string);
+}
+/** The caller's abort signal fired while waiting (Ctrl-C in a terminal,
+ *  viewer shutdown). The ticket has already been withdrawn. */
+declare class QueueWaitAbortedError extends UnotestError {
+    constructor(ticketName: string);
+}
+/** A queue directory that cannot be created / written. Distinct from a
+ *  full queue: this one never resolves by waiting. */
+declare class QueueUnavailableError extends UnotestError {
+    constructor(root: string, cause: string);
+}
+
+interface TarEntry {
+    /** Relative posix path inside the archive. */
+    path: string;
+    content: Uint8Array;
+    /** Only the exec bit travels. The rest of a file's mode is the packing
+     *  machine's umask, and a suite that hashes differently because of the
+     *  umask it was packed under would break the "same tree, same id"
+     *  promise the whole upload path rests on. */
+    executable?: boolean;
+}
+/** Everything a member path must be before it is written or restored.
+ *
+ *  Rejected on BOTH sides on purpose: the packing check is a courtesy to
+ *  whoever is packing, the unpacking check is the one that stops an
+ *  uploaded archive from writing outside the directory it was given. */
+declare function assertSafeBundlePath(path: string): void;
+/** Serialise entries into a tar archive. Entries are sorted by path and
+ *  duplicates are refused: an archive with two members of the same name
+ *  restores to whichever came last, which makes the id of a bundle depend
+ *  on the order somebody happened to walk a directory. */
+declare function writeTar(entries: readonly TarEntry[]): Buffer;
+/** Restore entries from a tar archive.
+ *
+ *  Strict by design: only plain files, only the ustar magic, only paths
+ *  that stay inside the bundle. A member this refuses is not a member we
+ *  would know what to do with — a symlink in an uploaded archive is a way
+ *  to make the next reader follow a path off the box. */
+declare function readTar(data: Uint8Array): TarEntry[];
+
+type BundleFile = TarEntry;
+/** FULL content hash of a payload: sha256 over the sorted list of
+ *  (path, exec bit, content hash). Two trees with the same files have
+ *  the same hash no matter how they were walked, compressed or
+ *  transported. This is the integrity anchor — the manifest carries it
+ *  whole, and the box verifies the WHOLE of it after unpacking. */
+declare function computeTreeSha256(files: readonly BundleFile[]): string;
+/** The short HANDLE derived from the full hash: a directory name, a URL
+ *  segment, a run-manifest field. Identity and integrity are the full
+ *  sha256; this is only how humans and paths refer to it. */
+declare function computeBundleId(files: readonly BundleFile[]): string;
+/** What the manifest carries that the files themselves cannot say. */
+interface BundleProvenance {
+    createdAt: number;
+    webVersion: string;
+    author?: string;
+    git?: BundleGitInfo;
+    /** Names of the externals the suite declares. The files that hold them
+     *  never travel (their values belong to an environment), so the names
+     *  ride in the manifest — that is what lets a box say at upload which
+     *  of them it cannot supply. */
+    declaredVariables?: readonly string[];
+}
+interface PackedBundle {
+    bundleId: string;
+    manifest: BundleManifest;
+    /** gzipped tar, ready to be written to a file or POSTed. */
+    archive: Buffer;
+}
+declare function packBundle(files: readonly BundleFile[], provenance: BundleProvenance): PackedBundle;
+interface UnpackedBundle {
+    manifest: BundleManifest;
+    /** The suite, without the manifest. */
+    files: BundleFile[];
+}
+interface UnpackOptions {
+    /** Cap on the UNPACKED size, enforced while decompressing. Without one
+     *  a 1 MB upload can expand into a full disk. */
+    maxBytes?: number;
+}
+declare function unpackBundle(archive: Uint8Array, options?: UnpackOptions): UnpackedBundle;
+
+/** The archive is not a readable bundle: broken gzip, a truncated tar, a
+ *  header whose checksum does not add up, a member that is not a plain
+ *  file. Everything a box refuses BEFORE writing anything to disk. */
+declare class BundleFormatError extends UnotestError {
+    constructor(reason: string, context?: Record<string, unknown>);
+}
+/** A member path a bundle may not carry.
+ *
+ *  This is a security boundary, not tidiness: an archive is written by
+ *  whoever uploaded it, and `../../etc/cron.d/x` or an absolute path in a
+ *  tar header is the oldest way to make an unpacker write outside the
+ *  directory it was told to. Paths are checked when PACKING and again
+ *  when UNPACKING — the second check is the one that matters, because the
+ *  first ran on the uploader's machine. */
+declare class BundlePathError extends UnotestError {
+    constructor(path: string, reason: string);
+}
+/** The archive unpacks to more than the caller allowed.
+ *
+ *  A separate error because the answer is different: a malformed archive
+ *  is never worth retrying, an oversized one means "split the suite or
+ *  raise the box's limit". It is also the defence against a small upload
+ *  that expands to fill a disk — the limit is enforced DURING
+ *  decompression, not after. */
+declare class BundleTooLargeError extends UnotestError {
+    constructor(limitBytes: number);
+}
+/** The archive's content does not hash to the id its manifest claims.
+ *  Either the upload was truncated or something rewrote it in flight; in
+ *  both cases the box must not store it under an id that means something
+ *  else — the id is what every ticket, symlink and cache key refers to. */
+declare class BundleIdMismatchError extends UnotestError {
+    constructor(claimed: string, computed: string);
+}
+
+export { type ArtifactRedactor, type BundleFile, BundleFormatError, BundleIdMismatchError, BundlePathError, type BundleProvenance, BundleTooLargeError, type DebugCommandsWatcherDeps, type DebugControlTarget, type DebugWatcherLogger, type E2EFlags, FsRunQueue, type FsRunQueueOptions, type GitignoreUpdate, type GlobalPoolOptions, type IRunArtifactWriter, JsonlRunArtifactWriter, type JsonlRunArtifactWriterOptions, type PackedBundle, QueueTicketLostError, QueueUnavailableError, QueueWaitAbortedError, type RunHeartbeat, type RuntimeExecState, type RuntimeInspection, type RuntimeInspectionInput, type RuntimeStateExtra, type RuntimeStateWriter, type RuntimeStateWriterDeps, type TarEntry, UnotestError, type UnpackOptions, type UnpackedBundle, type WriteManifestInput, type WriteRunSourcesInput, appendUniqueLines, applyEnvLayers, assertSafeBundlePath, buildRuntimeInspection, computeBundleId, computeTreeSha256, createRunArtifactWriter, createRuntimeStateWriter, currentLocation, extractCallStack, globalPoolFromEnv, leaseEnv, levenshteinDistance, packBundle, parseE2EFlags, readDebuggerBreakpoints, readEnvFile, readEnvLayers, readTar, startDebugCommandsWatcher, startRunHeartbeat, suggestClosest, toProtocolRuntimeState, unpackBundle, writeRunManifest, writeRunSources, writeTar };