@zq-silk/yui 0.6.4 → 0.6.6

package/README.md

@@ -21,7 +21,7 @@ yui setup
 yui doctor
 ```
 
-`setup` is interactive. It detects installed Agent CLIs, asks which Agents to configure, selects the default and Operator Agent, and probes each selected CLI for its current models. It configures the Leader and Operator, then explains that the global Worker configuration is copied into new Task Roles and asks whether Worker should reuse Leader or be configured separately. Model selection is followed by that model's supported reasoning efforts. Setup also confirms the Project workspace outside Yui home and offers shell-completion setup. The picker includes the native CLI default and a custom-value option. Running setup again preserves existing Tasks, Roles, and the installation's Project workspace while allowing safe configuration changes.
+`setup` is interactive. It detects installed Agent CLIs, asks which Agents to configure, selects the default and Operator Agent, and probes each selected CLI for its current models. It configures the Leader and Operator, then explains that the global Worker configuration is copied into new Task Roles and asks whether Worker should reuse Leader or be configured separately. Model selection is followed by that model's supported reasoning efforts. Setup also confirms the Project workspace outside Yui home and offers shell-completion setup. The picker includes the native CLI default and a custom-value option. Running setup again preserves existing Tasks, Roles, and the installation's Project workspace while allowing safe configuration changes. A successful setup ensures the current Home's detached Controller is running before it returns.
 
 Model and effort are per-Agent Role settings, so Operator, Leader, and the global Worker can use different values even when they share an Agent CLI. Interactive Role flows validate those settings against the selected Agent runtime. Worker Profile model and effort fields are provider-neutral child-execution hints and therefore remain explicit, scriptable values rather than Agent capability selections.
 
@@ -737,6 +737,12 @@ hiding the resources that remain. Use `--all` to include discovered Yui homes.
 
 `controller restart` replaces the Controller process and its scheduler/socket services with the currently installed Yui version. It does not stop or restart managed tmux/Agent sessions.
 
+Successful `setup`, `upgrade`, and `update` commands ensure that the current
+Home has a running Controller, starting one when the Home was previously idle.
+Read-only commands and `upgrade --dry-run` do not start a Controller. `update`
+also replaces an already-running Controller only after the new binary passes its
+health checks.
+
 Its recovery reconciliation runs every 120 seconds by default. Normal durable state changes enqueue a Task, Role, or Operator key and return immediately; keys received in the same fixed 100 ms window trigger one non-overlapping targeted pass. Operator presentation has an independent lane, so a blocked Task workspace operation cannot delay a user question. Periodic Git/worktree work is limited to Tasks with durable Task-mailbox work, while active Role liveness uses one tmux inventory. Agent Driver Hooks write exact-fenced observations to the durable runtime inbox without starting or waiting for the Controller. A terminal Turn observation gives a legal yield/input/completion two seconds to win before a forgotten Run fails its workflow contract. Durable mailboxes freeze the current batch while new signals merge into the next batch. Task-orchestration failures retain the exact Controller-owned processing batch for two bounded fast retries and later periodic recovery; a successful retry completes that batch before newer pending work is claimed. Recommended InputRequest and pending Turn deadlines share one nearest-deadline selector and therefore do not wait for the recovery interval. Explicit `task reconcile` still requests an immediate recovery pass. The retained loop is:
 
 1. dispatch pending Leader wakes whose Task workspaces are already ready;
@@ -821,8 +827,10 @@ preflight does not create or validate a staged Home and is not
 stops the exact old Controller PID. Current and compatible-old Homes use the
 no-Home-mutation fast path; migration-required Homes first require a clear
 offline Run/Session/lifecycle inventory, then use the timestamped-backup switch.
-Both paths promote the binary only after an exact PID-fenced old-Controller stop
-and run a new-binary health check before the authenticated replacement starts.
+Both paths run a new-binary health check before the authenticated Controller
+handoff. When an old Controller exists, it is stopped with an exact PID fence;
+when no old Controller existed, the same verified new Controller is started
+before the update reports success.
 Yui promotes the **same artifact it staged**
 (binary activation pins the exact staged version, never a second bare `@latest`);
 the staged version must be a **concrete semver** — a `latest`/dist-tag sentinel,

package/dist/cli/updateOrchestrator.js

@@ -223,12 +223,15 @@ function activateAndVerify(ports, staged, home, storageBackupPath, lifecycle, pa
         };
         return restoreBeforeSwitchOrReport(ports, home, lifecycle, storageBackupPath, failure);
     }
-    if (lifecycle?.wasRunning === true) {
+    if (lifecycle?.ensureRunning === true) {
         try {
             ports.startController(home);
         }
         catch (error) {
             const unknownActive = isUnknownActiveControllerFailure(error);
+            const startFailureAction = lifecycle.wasRunning
+                ? "The Home was not migrated. Keep writes quiesced and restore the previously running Controller identity before retrying."
+                : "The Home was not migrated. Keep writes quiesced and start the replacement Controller after verifying the activated binary.";
             const failure = {
                 outcome: "aborted",
                 phase: "post-verify",
@@ -237,7 +240,7 @@ function activateAndVerify(ports, staged, home, storageBackupPath, lifecycle, pa
                 action: unknownActive
                     ? unknownActiveControllerAction(home, storageBackupPath)
                     : storageBackupPath === undefined
-                        ? "The Home was not migrated. Keep writes quiesced and restore the previously running Controller identity before retrying."
+                        ? startFailureAction
                         : postSwitchRecoveryAction(home, storageBackupPath),
                 recoverable: false,
                 version: staged.version,
@@ -276,8 +279,9 @@ function captureControllerLifecycle(ports, version, home) {
         ports.startController,
         ports.restoreController
     ].some((port) => port !== undefined);
-    if (!supplied)
-        return { lifecycle: { wasRunning: false, stopped: false } };
+    if (!supplied) {
+        return { lifecycle: { ensureRunning: false, wasRunning: false, stopped: false } };
+    }
     if (ports.controllerStatus === undefined
         || ports.stopController === undefined
         || ports.startController === undefined
@@ -315,8 +319,9 @@ function captureControllerLifecycle(ports, version, home) {
             version
         };
     }
-    if (!status.running)
-        return { lifecycle: { wasRunning: false, stopped: false } };
+    if (!status.running) {
+        return { lifecycle: { ensureRunning: true, wasRunning: false, stopped: false } };
+    }
     if (!isPositivePid(status.pid)) {
         return {
             outcome: "aborted",
@@ -365,6 +370,7 @@ function captureControllerLifecycle(ports, version, home) {
     }
     return {
         lifecycle: {
+            ensureRunning: true,
             wasRunning: true,
             stopped: true,
             identity: status.identity

package/dist/cli.js

@@ -177,6 +177,10 @@ export async function main() {
         };
         validateSetupInvocation(args.slice(1), setupIo);
         const output = await runSetupCommand(args.slice(1), process.env, new NodeCommandExecutor(), setupIo);
+        // A successful setup leaves the Home ready for normal Yui work. Start the
+        // detached per-Home Controller even when setup began with no Controller;
+        // read-only commands and failed setup still remain non-starting paths.
+        await ensureFileTaskController(home, { environment: process.env });
         const refresh = await refreshRunningFileTaskControllerEnvironment(home, openCompatibleFileTaskStore(home), process.env);
         emit(withControllerRefreshWarning(output, refresh, "Agent environment"));
         return;
@@ -220,6 +224,14 @@ export async function main() {
         const result = await runUpgradeCommand(args.slice(1), home, process.env.YUI_UPDATE_EXTERNALLY_QUIESCED === "1"
             ? { controllerLifecycle: "externally-quiesced" }
             : {});
+        // Public execute upgrades leave the Home operational even when no
+        // Controller existed before the command. Dry-run and the staged updater's
+        // externally-quiesced preflight must remain read-only/lifecycle-neutral.
+        if (args.length === 1
+            && result.exitCode === 0
+            && process.env.YUI_UPDATE_EXTERNALLY_QUIESCED !== "1") {
+            await ensureFileTaskController(home, { environment: process.env });
+        }
         process.exitCode = result.exitCode;
         emit(result.output, false, result.data);
         return;

package/dist/controller/agentRuntimeObserver.js

@@ -1,6 +1,8 @@
 import { createHash } from "node:crypto";
 import { builtinAgentDriverRegistry } from "../runtime/builtinAgentDrivers.js";
 import { createRuntimeObservation, runtimeObservationFenceMatches, runtimeObservationFromTaskEvent } from "../runtime/runtimeObservation.js";
+const DEFAULT_MAX_CONCURRENT_SAMPLES = 8;
+const MAX_CONCURRENT_SAMPLES = 64;
 /**
  * Controller-owned, provider-independent sampler. Drivers own source parsing;
  * this component owns active-Run discovery, cursor lifetime, canonical event
@@ -11,11 +13,13 @@ export class AgentRuntimeObserver {
     inbox;
     drivers;
     #states = new Map();
+    #maxConcurrentSamples;
     #sequence = 0;
-    constructor(store, inbox, drivers = builtinAgentDriverRegistry()) {
+    constructor(store, inbox, drivers = builtinAgentDriverRegistry(), options = {}) {
         this.store = store;
         this.inbox = inbox;
         this.drivers = drivers;
+        this.#maxConcurrentSamples = sampleConcurrency(options.maxConcurrentSamples);
     }
     async sample(now = new Date()) {
         const active = this.activeSources();
@@ -25,7 +29,9 @@ export class AgentRuntimeObserver {
                 this.#states.delete(key);
         }
         const dirty = new Set();
-        await Promise.all(active.map(async ({ key, fence, source, freshSession, persistedState }) => {
+        const sequenceBase = this.#sequence;
+        this.#sequence += active.length;
+        await forEachConcurrent(active, this.#maxConcurrentSamples, async ({ key, fence, source, freshSession, persistedState }, index) => {
             const existingState = this.#states.get(key);
             // Cursor state is intentionally process-local, but the latest canonical
             // usage/activity baseline is durable. Rehydrate it after Controller
@@ -49,7 +55,10 @@ export class AgentRuntimeObserver {
             }
             state.cursor = sample.cursor;
             const at = now.toISOString();
-            const sequence = this.#sequence++;
+            // Reserve sequence numbers in source-key order before sampling. Provider
+            // latency can change completion order without changing observation
+            // identity or the canonical sequence assigned to a source.
+            const sequence = sequenceBase + index;
             if (existingState === undefined && freshSession && state.usage === undefined) {
                 const zero = Object.freeze({ inputTokens: 0, outputTokens: 0 });
                 this.inbox.enqueueObservation(createRuntimeObservation({
@@ -114,28 +123,52 @@ export class AgentRuntimeObserver {
             if (sample.activityId !== undefined)
                 state.activityId = sample.activityId;
             this.#states.set(key, state);
-        }));
-        return Object.freeze([...dirty].sort());
+        });
+        return Object.freeze([...dirty].sort(numericCompare));
     }
     activeSources() {
         const result = [];
-        for (const task of this.store.listTasks()) {
-            if (task.status !== "active")
-                continue;
-            const observations = this.store.listEvents(task.id)
-                .map(runtimeObservationFromTaskEvent)
-                .filter((value) => value !== null);
+        const indexedTaskIds = this.store.listActiveTaskIds?.();
+        // FileTaskStore remains a development/compatibility fallback. The normal
+        // Controller store is SQLite and must discover only its indexed hot set.
+        const activeTasks = indexedTaskIds === undefined
+            ? this.store.listTasks().filter((task) => task.status === "active")
+            : [...new Set(indexedTaskIds)]
+                .sort(numericCompare)
+                .map((taskId) => this.store.getTask(taskId))
+                .filter((task) => (task !== null && task.status === "active"));
+        for (const task of activeTasks) {
+            // A Task still incurs one O(E) event projection. Group those observations
+            // by Run and sort each group once so every active Run can reuse the same
+            // ordered slice instead of repeatedly filtering/sorting all E events.
+            const observationsByRunId = new Map();
+            for (const event of this.store.listEvents(task.id)) {
+                const observation = runtimeObservationFromTaskEvent(event);
+                const runId = observation?.fence.runId;
+                if (observation === null || runId === undefined)
+                    continue;
+                const grouped = observationsByRunId.get(runId);
+                if (grouped === undefined) {
+                    observationsByRunId.set(runId, [observation]);
+                }
+                else {
+                    grouped.push(observation);
+                }
+            }
+            for (const observations of observationsByRunId.values()) {
+                observations.sort(compareObservations);
+            }
             for (const run of this.store.listAgentRuns(task.id)) {
                 if (run.status !== "active"
                     || this.store.getActiveAgentRun(task.id, run.roleName)?.id !== run.id)
                     continue;
+                const observations = observationsByRunId.get(run.id) ?? [];
                 const accepted = observations
                     .filter((observation) => observation.kind === "turn.accepted"
                     && observation.fence.runId === run.id
                     && observation.fence.roleName === run.roleName
                     && observation.fence.agentId === run.effective.agentId
                     && observation.payload.observerSource !== undefined)
-                    .sort(compareObservations)
                     .at(-1);
                 const source = accepted?.payload.observerSource;
                 if (accepted === undefined || source === undefined
@@ -150,19 +183,23 @@ export class AgentRuntimeObserver {
                 }
                 const fence = accepted.fence;
                 const exact = observations
-                    .filter((observation) => runtimeObservationFenceMatches(fence, observation.fence))
-                    .sort(compareObservations);
+                    .filter((observation) => runtimeObservationFenceMatches(fence, observation.fence));
                 const persistedUsage = exact.filter((observation) => (observation.kind === "activity.observed"
                     && observation.payload.usage !== undefined)).at(-1);
                 const persistedHealth = exact.filter((observation) => (observation.kind === "observer.health"
                     && observation.payload.sourceId === source.sourceId)).at(-1);
                 result.push(Object.freeze({
                     key: JSON.stringify([
+                        fence.taskId,
+                        fence.roleName,
+                        fence.runId,
+                        fence.agentId,
                         fence.driverId,
+                        fence.launchId,
                         fence.sessionGenerationId,
                         fence.nativeSessionId,
                         fence.nativeTurnId,
-                        fence.runId,
+                        fence.receiptId,
                         source.sourceId
                     ]),
                     fence,
@@ -187,8 +224,31 @@ export class AgentRuntimeObserver {
                 }));
             }
         }
-        return result;
+        return Object.freeze(result.sort((left, right) => numericCompare(left.key, right.key)));
+    }
+}
+async function forEachConcurrent(values, limit, visit) {
+    let nextIndex = 0;
+    const workers = Math.min(limit, values.length);
+    await Promise.all(Array.from({ length: workers }, async () => {
+        while (nextIndex < values.length) {
+            const index = nextIndex;
+            nextIndex += 1;
+            await visit(values[index], index);
+        }
+    }));
+}
+function sampleConcurrency(value) {
+    const resolved = value ?? DEFAULT_MAX_CONCURRENT_SAMPLES;
+    if (!Number.isSafeInteger(resolved)
+        || resolved < 1
+        || resolved > MAX_CONCURRENT_SAMPLES) {
+        throw new Error(`Agent runtime observer maxConcurrentSamples must be between 1 and ${MAX_CONCURRENT_SAMPLES}.`);
     }
+    return resolved;
+}
+function numericCompare(left, right) {
+    return left.localeCompare(right, undefined, { numeric: true });
 }
 function compareObservations(left, right) {
     return left.receivedAt.localeCompare(right.receivedAt)