@jc_stack/ez-agents 0.1.0-beta.51 → 0.1.0-beta.58

package/CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## 0.1.0-beta.58
+
+Versions 0.1.0-beta.53 through 0.1.0-beta.57 were used only by private QA candidates; this release follows the newest installed candidate (0.1.0-beta.58.rc.1) so it can upgrade in place.
+
+- Add an owner chat/application fallback chain (`ezenciel-agents-ai fallbacks`), captured when work is queued, with the scheduled-fallback safeguards: advance only on typed unavailability, rejection before work or a usage limit, never after cancellation, access denial, revoked authority or an uncertain outcome. Same-CLI logins keep the turn's model and effort, and each login keeps its own native session per conversation.
+- Add opt-in per-attempt run telemetry (`ezenciel-agents-ai telemetry on`): reported model, deduplicated token counts, tool counts/durations and error categories with labelled coverage gaps, never prompts, tool data, credentials or costs.
+- Make schedule edits safe for queued and running work. `edit` now requires `--expected-revision` and rejects a stale revision. It applies only while the task is idle, or with `--stage` saves the change for future occurrences while queued and running ones finish on their captured settings, including fallbacks and quota retries. Edits, dispatch, trigger, pause, resume and remove serialize on a per-task lock, so a due occurrence cannot land between the check and the write. Edit needs the running relay. A root invocation of the schedule CLI runs as the control directory's owner, so its definitions stay readable by the relay and it cannot follow agent-planted symlinks.
+- Hold new native turn admission while a completed or rolled-back core update drains for supervisor reload. Retain queued requests and reopen only after the replacement worker is ready.
+- Preserve scheduled fallback handling when a compatible relay upgrade adopts a running native session, including continuation after a terminal usage limit without replaying its work.
+- Reuse the last successful same-model native login across scheduled tasks and relay restarts; rotate through configured logins and skip known exhausted scopes until their provider reset.
+- Keep scheduled work queued when every saved CLI login reaches its usage limit, instead of failing on each tick. Known resets wake the occurrence; unknown resets back off from 5 minutes to hourly. Codex reports native rate limits, and Claude credits-required responses wait for an explicit retry. `ezenciel-agents-schedule retry [RUN_ID]` and `/retry_quota [RUN_ID]` clear the relevant cooldowns after a manual reset or added credits.
+- Add `ezenciel-agents-schedule running [--json]`, a read-only view of queued and running current-owner foreground turns and scheduled runs with their engine, timing and schedule origin. It exports no prompts, messages or native output.
+- Restore isolated Codex chat replies: the isolated executor derives the Codex sandbox setting from its trusted deployment environment, never from an incoming job.
+- Keep unfinished Claude background work alive in print mode until completion or explicit cancellation, instead of being killed after 10 idle minutes.
+
+Beta limits: live fresh-host/reboot acceptance and live installed-agent QA of the chat fallback chain, run telemetry and safe schedule edits remain pending; they are covered by synthetic relay, CLI and container tests. A Codex chat setup advances only when its CLI is unavailable, so put Codex last in a chat chain. On its first turn, a fallback login's linked session does not see the conversation's earlier native history. Run telemetry and quota-waiting runs live in relay memory and reset on restart; saved-login cooldowns persist. Callers of `ezenciel-agents-schedule edit` must now pass `--expected-revision`, and edit needs the running relay. Publication does not establish an installed-agent upgrade or authorize provider/domain writes.
+
+## 0.1.0-beta.52
+
+- Restore isolated executor startup with existing named volumes and a private, root-owned read-only container binding. Host file ownership and permissions remain unchanged.
+- Start a missing isolated executor when native requests are queued, while retaining the live-run drain and single-worker ownership checks.
+
+
 ## 0.1.0-beta.51
 
 - Keep native runs alive during handoff-compatible core upgrades. The retiring executor finishes existing work while its replacement accepts new requests; the relay adopts runs without replay and preserves schedule holds, cancellation and settled delivery receipts. Isolated agents migrate once while idle to a separate, unprivileged executor container; later compatible upgrades preserve its running generation. Older runtimes without the handoff contract retain the full native-work drain.

package/README.md

@@ -333,4 +333,7 @@ The core `ezenciel-agents-schedule` CLI accepts instruction text for one-off dat
 intervals and timezone-aware cron schedules. `create --now` delegates a task to a
 separate CLI session queued with owner turns in the bound workspace. Long work has
 no production wall-clock timeout; goals and subagents remain native executor
-features. See [scheduling, recovery and QA](docs/scheduling.md).
+features. Use `ezenciel-agents-schedule running` for a read-only view of queued
+and running foreground/scheduled work, or add `--json` for an agent-readable
+snapshot. Native child status and tool activity are unavailable; old or missing
+telemetry does not imply idle or complete. See [scheduling, recovery and QA](docs/scheduling.md).

package/docker/isolated-executor-smoke.mjs

@@ -0,0 +1,35 @@
+// Synthetic private binding and existing-volume acceptance; no provider access.
+import * as fs from 'node:fs/promises';
+import path from 'node:path';
+import {tmpdir} from 'node:os';
+import {randomUUID} from 'node:crypto';
+import {execute} from '../src/updates/runtime.mjs';
+import {executorBindingArchive,executorVolumes} from '../src/updates/supervisor.mjs';
+const root=await fs.mkdtemp(path.join(tmpdir(),'ez-executor-smoke-'));
+const project='ez-executor-smoke-'+randomUUID().slice(0,8),volume=project+'-existing';
+const file=path.join(root,'compose.json'),image=process.env.EZ_RELAY_IMAGE||'ezenciel-agents:local';
+const binding={agents:[{workspace:'/synthetic',binDir:'/app/bin'}]};
+const code=`const fs=require('fs'),assert=require('assert/strict');const p='/run/executor.json',s=fs.statSync(p),d=fs.statSync('/run');assert.equal(process.getuid(),1001);assert.equal(s.uid,0);assert.equal(s.gid,1000);assert.equal(s.mode&511,288);assert.equal(d.uid,0);assert.equal(d.mode&18,0);assert.deepEqual(JSON.parse(fs.readFileSync(p)),${JSON.stringify(binding)});for(const fn of [()=>fs.writeFileSync(p,'changed'),()=>fs.chmodSync(p,384),()=>fs.unlinkSync(p)])assert.throws(fn,e=>['EACCES','EPERM','EROFS'].includes(e.code));assert.equal(fs.readFileSync('/ipc/marker','utf8'),'retained');console.log('Private readonly binding and retained volume passed');`;
+const worker={image,user:'1001:1000',network_mode:'none',cap_drop:['ALL'],security_opt:['no-new-privileges:true'],entrypoint:['node','-e',code],volumes:[{type:'volume',source:'ipc',target:'/ipc'}]};
+const args=['compose','--project-name',project,'--file',file];
+try {
+ await execute('docker',['volume','create',volume]);
+ await execute('docker',['run','--rm','--network','none','--user','0:0','--mount',`type=volume,source=${volume},target=/ipc`,'--entrypoint','node',image,'-e',"require('fs').writeFileSync('/ipc/marker','retained')"]);
+ await fs.writeFile(file,JSON.stringify({services:{executor:worker},volumes:executorVolumes({volumes:{ipc:{name:volume}}},worker)}),{mode:0o600});
+ await execute('docker',[...args,'create','--no-build','executor']);
+ const cid=(await execute('docker',[...args,'ps','-aq','executor'],{stdoutOnly:true})).trim();
+ await execute('docker',['cp','-',`${cid}:/run`],{input:executorBindingArchive(binding,1000)});
+ console.log(await execute('docker',['start','-a',cid]));
+ const exit=(await execute('docker',['inspect','--format','{{.State.ExitCode}}',cid],{stdoutOnly:true})).trim();
+ if(exit!=='0')throw Error('Executor binding acceptance failed');
+ await execute('docker',[...args,'down']);
+ const retained=(await execute('docker',['volume','inspect','--format','{{.Name}}',volume],{stdoutOnly:true})).trim();
+ if(retained!==volume)throw Error('External volume identity changed');
+ // Rejected archive input must fail cleanly, with no uncaught stdin EPIPE.
+ await execute('docker',['cp','-',`${project}-missing:/run`],{input:Buffer.alloc(1024*1024)}).then(()=>{throw Error('Missing container copy succeeded');},()=>{});
+ console.log('PASS: unprivileged executor reads only its private binding; writes, chmod and unlink fail; external volume preserved.');
+} finally {
+ await execute('docker',[...args,'down']).catch(()=>{});
+ await execute('docker',['volume','rm',volume]).catch(()=>{});
+ await fs.rm(root,{recursive:true,force:true});
+}

package/docs/architecture/ai-selection.md

@@ -2,7 +2,10 @@
 
 `menu.ts` owns Telegram controls; `ai.ts` projects installed-client metadata.
 `ControlStore` stores saved presets, the next-conversation default and current choice.
-No provider SDK, model fallback, history migration, or second execution loop.
+No provider SDK, history migration, or second execution loop. The only model
+fallback is an owner-saved ordered chain captured with each turn and advanced by
+the relay's existing attempt loop on typed evidence
+([chat fallback chain](../docker-runtime.md#chat-fallback-chain)).
 A missing catalog entry rejects selection without changing the conversation.
 An unsupported effort reports the exact installed model's advertised levels.
 Missing selectable models do not establish that the active native session is
@@ -19,7 +22,8 @@ and native history; subsequent turns pass the new settings to the native engine.
 Changing client, provider or auth profile starts a separate native conversation.
 A conversation records its client and auth profile when created; launch rejects a
 choice for another profile, so a native session never resumes under another
-account. Provisioned profiles appear as additional catalog entries for their
+account; a fallback login uses its own linked native session for the
+conversation. Provisioned profiles appear as additional catalog entries for their
 client (see [auth profiles](../docker-runtime.md#auth-profiles)). Old bindings
 remain available so accepted work can finish and owners can reopen their chats.
 

package/docs/docker-runtime.md

@@ -286,6 +286,9 @@ On macOS host jobs, Seatbelt also denies sibling directories of the bound
 workspace (a tenant farm), while allowing explicitly shared and additional
 workspace grants and leaving `$HOME` readable for the host CLI.
 Codex then uses `danger-full-access` so it does not apply a nested Seatbelt.
+The isolated executor derives the Codex sandbox setting from its trusted
+deployment environment, never from an incoming job. Host workers and restricted
+tasks retain their existing native sandbox.
 
 ## Claude Code auth
 
@@ -338,6 +341,51 @@ the profile entries in Choose AI, or `ezenciel-agents-schedule create ...
 profile starts a new native conversation, and a run whose request names another
 profile than its captured choice is rejected before launch.
 
+## Chat fallback chain
+
+Chat and application turns can name an ordered chain after the selected AI, for
+example Claude default → three other Claude logins → Codex:
+
+```sh
+ezenciel-agents-ai fallbacks --fallback cli=claude,auth-profile=claude2 \
+  --fallback cli=claude,auth-profile=claude3 --fallback cli=claude,auth-profile=claude4 \
+  --fallback cli=codex,model=gpt-6.1-sol,effort=xhigh
+```
+
+An entry for the selected AI's CLI without provider, model or effort keeps the
+turn's own model and effort, so every role keeps its settings on other logins.
+Each queued turn captures the chain with its choice; later edits apply to later
+turns only. `fallbacks` alone shows the resulting order and `--clear` removes it.
+Profiles must be provisioned and models advertised before they are saved.
+
+The rules of [ordered setup fallback](scheduling.md#ordered-setup-fallback)
+apply: core advances only on typed evidence that a setup was unavailable or
+rejected before work, or stopped at its usage limit (the next setup is told the
+work may be partly applied). Access denial, other failures after work began,
+unknown outcomes, owner stop and revoked owner or application authority end the
+turn without replay. A login cooling down after a recorded quota rejection is
+skipped while another is available; when all are, they are tried in order.
+Codex chat turns report no typed usage-limit rejection, so a Codex setup
+advances only when its CLI is unavailable; put Codex last in the chain.
+Each login keeps its own native session for the conversation, created on first
+use and resumed on later turns; a session never resumes under another login. A
+fallback login's first turn does not see the conversation's earlier native
+history; durable context lives in the workspace.
+
+## Run telemetry
+
+`ezenciel-agents-ai telemetry on` (off by default) records, for every attempt
+started afterwards, the model the client reported, input/output/cache-read/
+cache-write token counts, tool call counts, errors and observed durations by
+tool name, and error categories. They appear in each attempt of
+`ezenciel-agents-schedule run RUN_ID`. Counts come from structured client
+events: repeated per-message usage is counted once, a client's cumulative total
+replaces partial sums, and `coverage`/`gaps` label anything missing or partial,
+such as a resumed Codex session whose total may include earlier turns. Prompts,
+model text, tool arguments and results, credentials and reasoning are never
+read into telemetry, and no cost or historical usage is derived. Records stay
+in relay memory with the run and reset on restart.
+
 ## OpenCode auth
 
 Isolated agents run the image OpenCode with `XDG_DATA_HOME` set to the

package/docs/releasing.md

@@ -22,6 +22,7 @@ separate release bot is not required.
    audit advances. Baseline CI does not validate a newly published native version.
    Managed installs and upgrades keep their existing native version selection.
    Main relay: `EZ_RELAY_IMAGE=ez-release-runtime node docker/smoke.mjs`.
+   Isolated executor: `EZ_RELAY_IMAGE=ez-release-runtime node docker/isolated-executor-smoke.mjs`.
    The WhatsApp Compose overlay has no separate relay image; verify its
    provider integration with the synthetic plugin smoke in `docs/plugins.md`.
 4. Create an artifact with `npm pack --ignore-scripts`. Inspect its file list,

package/docs/scheduling.md

@@ -13,6 +13,8 @@ ezenciel-agents-schedule create --name Research --now \
   --text '/goal Complete the authorized research objective. Save evidence, verify the outcome, and send the owner the result.'
 ezenciel-agents-schedule list
 ezenciel-agents-schedule runs
+ezenciel-agents-schedule running
+ezenciel-agents-schedule running --json
 ezenciel-agents-schedule trigger SCHEDULE_ID --key owner-smoke-20261003
 ezenciel-agents-schedule pause SCHEDULE_ID
 ezenciel-agents-schedule resume SCHEDULE_ID
@@ -20,13 +22,75 @@ ezenciel-agents-schedule remove SCHEDULE_ID
 ezenciel-agents-schedule cancel RUN_ID
 ```
 
-Use `--text-file` for longer instructions. `edit ID` replaces the complete schedule
-with a new revision; use the same trigger/name/text flags as `create`. It keeps
-the paused/enabled state: an edited paused task stays paused until `resume`.
+Use `--text-file` for longer instructions. `edit ID --expected-revision REVISION`
+replaces the complete schedule with a new revision; use the same trigger/name/text
+flags as `create`. `REVISION` is the `revision` printed by `show`, `list`,
+`create` or a previous `edit`. If the schedule changed since that revision, the
+edit is rejected and nothing is written: read it again and reapply the change.
+It keeps the paused/enabled state: an edited paused task stays paused until `resume`.
 An edit keeps a future pending occurrence when its trigger is unchanged;
 otherwise the edited rule begins at its next future occurrence. Editing does
 not replay missed work from an old start date. Use `--now` to request a new
 one-time run explicitly. `show` reports the pending occurrence used by dispatch.
+
+An edit never disrupts an occurrence that is already queued or running:
+
+- Without `--stage`, it applies only while the task is idle. When an occurrence
+  is queued (including a quota wait) or running, it is rejected with those run IDs
+  and nothing changes. Edit again after they finish, or `cancel RUN_ID` first.
+- With `--stage`, the new revision is saved for future occurrences. The queued or
+  running occurrences listed in `continuingRunIds` finish with the instructions,
+  engine, model, effort, fallbacks, delivery binding and revision they captured,
+  including fallback advancement and quota retries. The next occurrence of the new
+  revision waits for them, as usual for one task. A continuing occurrence that is
+  interrupted (or fails, for a `--when unreviewed-failures` schedule) holds the
+  new revision until the next explicit edit.
+
+The compare, idle check and write happen under the task's lock, which dispatch,
+`trigger`, `pause`, `resume`, `remove` and quota `retry` also take; a due occurrence therefore
+lands either before the edit (and is protected as above) or after it, on the new
+revision. Edit needs the running relay to read queued and running occurrences;
+an unavailable relay rejects the edit rather than assuming the task is idle.
+`list`, `show`, `create`, `pause`, `resume`, `remove` and script registration keep
+working with the relay stopped.
+
+Definitions are private files owned by the agent runtime user. A root invocation,
+such as `docker compose exec relay ezenciel-agents-schedule ...`, first switches to
+the user and group that own the control directory, so saved definitions stay
+readable by the relay and root never writes through paths the agent controls.
+
+`running` gives one read-only view of **queued and running current-owner
+foreground turns (Telegram/application) and scheduled runs** in the bound relay.
+It shows run IDs,
+schedule names/IDs/revisions, the saved or latest attempted engine/model, start and
+elapsed times, and the last recorded relay activity. `--json` returns the same
+sanitized metadata with `schemaVersion`, `observedAt`, `counts`, `coverage` and
+`runs`, suitable for asking the agent to summarize what is running. Other commands
+keep their existing JSON output. The relay must be reachable; an unavailable
+relay is an error, never an empty/idle result.
+
+Activity means a recorded queue/start, setup attempt or delivery enqueue time;
+it does not measure native tool progress. An old observation does not change a
+run's recorded status. Missing start/model/activity metadata stays unknown, and
+finished runs are omitted. Schedule names are available only when the saved
+definition still matches that run's revision. For ordered fallbacks, the latest
+non-skipped attempt is shown with its recorded outcome. Script runs have no
+engine/model.
+
+`origin` and `scheduledChildren` link scheduled work to its recorded originating
+request. An origin may already be finished; its status is unknown when its record
+is outside retained owner coverage. These links are request provenance, not a
+native subagent tree. Native child status and native tool activity are unavailable
+on the current adapters, including Codex, Codex desktop, Claude, Grok, OpenCode
+and Pi. Native sessions, provider quotas and process lists are not queried.
+Prompts, message text, script arguments and raw engine output are not exported.
+
+This view covers one relay's retained memory. Ordinary restarts drop the ledger;
+gracefully adopted host runs are the existing exception. Standalone manual
+CLI/desktop sessions, other installations and restricted external/isolated tasks
+are outside this view. Application request context is omitted. Future schedule
+occurrences that have not been queued remain in `list`/`show`.
+
 Intervals use `--every-seconds` (minimum 60), optional `--start`, and optional
 `--until`. Cron also supports start/end bounds. All absolute timestamps require
 an explicit offset. `--now` starts on the next relay tick, not synchronously.
@@ -92,20 +156,33 @@ error, unavailable model or bad request), including a missing, expired or revoke
 login (typed authentication failure or HTTP401). Native API-error metadata does
 not count as model/tool work. It stops, without replay, on access
 denial (HTTP403, account or organization restriction), cancellation, relay shutdown,
-a schedule edit, a policy or unknown error, an interrupted transport, or any
+schedule removal, a policy or unknown error, an interrupted transport, or any
 other failure after work began. Evidence comes from structured events only: Codex
 native-session `codexErrorInfo` and Claude `stream-json` error fields (scheduled
 Claude runs use `--output-format stream-json`; the host forwards only a compact
 startup summary). Other engines advance only when their CLI is unavailable.
 
-The one failure after work began that continues is a Claude usage limit the run
-ended on (for example "You've hit your session limit"): a typed quota rejection
-with no model, tool or hook activity after it. A native session cannot be resumed
+After work begins, only a typed Codex or Claude usage limit, or a Claude
+credits-required rejection, permits continuation. The run must have ended on
+that rejection, with no subsequent model, tool or hook activity. A native session cannot be resumed
 under another login's configuration home, so the next eligible setup starts a fresh
 session with the saved prompt plus one factual line naming the run, the stopped
 setup, its native session and reset time, and that its work may be partly applied.
 The agent reconciles from its own records; core replays nothing else.
 
+These same completion and fallback rules apply after a graceful relay upgrade
+adopts a running native session. Adoption attaches to that session; only an
+eligible terminal failure starts the next saved setup.
+
+For consecutive primary choices that differ only by native login profile, later
+scheduled runs start with the last successfully completed login and rotate through
+the remaining saved logins if it becomes unavailable. The existing owner control
+state retains that preference and typed quota reset times across relay restarts.
+Other tasks with saved fallbacks for those logins skip an exhausted scope until its reset;
+unknown reset times use progressively spaced checks (5, 15, 30, then 60 minutes). A concurrent older run's
+success does not erase a future reset. Saved models, effort, heterogeneous fallback
+priority and chat selection are preserved; no unconfigured login is added.
+
 A login or quota rejection skips the remaining setups in the same scope: the native
 client and login profile (`claude`, `codex` including desktop and custom Codex
 providers), `grok`, or the upstream provider for OpenCode/Pi models. A named
@@ -113,17 +190,47 @@ profile selects a separate owner-provisioned account and is eligible for its own
 attempt; changing only the model in the same profile does not reset its scope.
 Do not provision aliases of the same account to obtain more allowance. Ez does
 not infer account independence from credentials or promise that a shared
-organization's allowance is separate. Each setup runs at most once per occurrence.
+organization's allowance is separate. Each setup runs at most once per availability check; quota waits retain the earlier attempts.
 
 Every attempt keeps the same run ID, occurrence, schedule revision, workspace,
-delivery binding and scheduled concurrency slot, in a fresh native session.
+delivery binding, in a fresh native session. Quota waits release the scheduled concurrency slot.
 `run RUN_ID` shows `attempts`: each setup, quota scope, outcome, failure category
 (`cli-unavailable`, `login-unavailable`, `quota`, `provider-rejected`, `access-denied`,
-`quota-after-work-began`, `after-work-began`, `uncertain`, or `same-login-scope`/`same-quota-scope`
-for a skipped setup), exit code, reported reset time and the session ID Ez assigned. If every setup is unavailable the run fails with
-reason `setups-unavailable`, a summary of each setup, and an owner notice on
-Telegram-bound tasks. Core never purchases credits, enables overage or consumes
-reset credits; the next regular occurrence runs normally.
+`quota-after-work-began`, `credits-required`, `credits-required-after-work-began`, `after-work-began`, `uncertain`, or `same-login-scope`/`same-quota-scope`
+for a skipped setup), exit code, reported reset time and the session ID Ez assigned.
+
+When all saved logins are quota-blocked, the same occurrence stays `queued` with
+`quotaWait`. A known reset schedules its next availability check. If it passed during native
+shutdown, one prompt probe precedes ordinary backoff. Unknown resets
+back off per login to an hourly maximum. A credits-required rejection waits for
+explicit retry. Other unavailable setups still fail with `setups-unavailable`.
+Waiting consumes no native process or scheduled execution slot. The scheduler
+keeps one pending occurrence per task, so repeated ticks do not accumulate copies.
+Waiting occurrences share one probe per login until native work is accepted.
+`/status` and the queued/running-work view show waiting and the next check.
+
+After adding credits or manually resetting a provider limit, use either:
+
+```sh
+ezenciel-agents-schedule retry           # wake current queued quota waits
+ezenciel-agents-schedule retry RUN_ID    # retry this waiting or prior quota-failed occurrence
+```
+
+Telegram: `/retry_quota` or `/retry_quota RUN_ID` (`/retry-quota` also works).
+The command works without an inference turn. It clears the cooldowns for the
+occurrence's saved login scopes and requests a new availability check. It never
+adds a login, changes the model or buys credits. A specific prior run is eligible
+only when its failure was entirely quota-related, its schedule is current and
+enabled, and no other occurrence of that task is queued or running. Cancelled,
+interrupted, edited, foreign-owner and other failed runs cannot be retried this way.
+`/retry` continues to handle failed incoming messages.
+
+Prior attempts and stopped native-session references remain in the same run.
+After partial work, the engine receives the existing factual continuation line
+and reconciles effects from its own records. Core reconstructs no history.
+Cooldowns persist in owner control state; waiting run records follow the existing
+relay-memory retention contract and do not survive a restart. Core never purchases
+credits, enables overage or consumes reset credits.
 
 ## Registered script schedules
 
@@ -249,8 +356,9 @@ Schedule definitions and cursors use protected atomic JSON files. A restart
 between queue creation and cursor persistence does not duplicate an occurrence.
 An overdue one-time job runs on return. Missed recurring occurrences coalesce into
 one pending run; future runs resume at the next eligible time. A paused queued
-occurrence waits for resume. Editing/removing a schedule invalidates its old
-queued occurrence; already-running work continues until explicitly cancelled.
+occurrence waits for resume. Removing a schedule invalidates its queued
+occurrence; already-running work continues until explicitly cancelled. An edit
+never invalidates queued or running work (see `edit` above).
 
 `/stop` stops all active work; `cancel RUN_ID` stops one background task. `/cancel`
 clears queued work. Pause/remove a recurring schedule to prevent future runs.

package/docs/upgrades.md

@@ -16,7 +16,7 @@ automatic channel for core and plugins without a saved policy. Existing explicit
 stable or manual policies are preserved. The owner may select either per target.
 The main target and installed plugins version independently.
 
-Handoff-compatible core upgrades build while native work continues and pause admission only for activation. The current native executor retires from claiming new work; the replacement claims new requests while the retiring generation finishes its existing runs. The relay adopts those runs without replay, preserving their schedule holds, cancellation and settled delivery receipts. The supervisor reloads only after native work is idle, so OS service process-group cleanup cannot kill a retiring generation. Plugin upgrades build while native work continues. Activation closes the existing registry admission lock, drains in-flight plugin command leases, and replaces the plugin under that lock. New native plugin calls wait for admission; an outdated revision fails before provider execution and must be rediscovered. No admitted provider operation is replayed. The shared broker and native sessions remain running. Older runtimes without the handoff contract retain the full native-work drain. Isolated agents migrate once while idle to a separate, unprivileged executor container using the same file transport; later relay/runtime upgrades preserve that executor generation. No native CLI or provider credentials move to the host. A plugin update also retains that drain if an operator replaced the broker’s single-file host binding (for example to approve new network pins), because that case still requires broker recreation.
+Handoff-compatible core upgrades build while native work continues; activation and the later supervisor reload hold new turn admission. The current native executor retires from claiming new work; the replacement claims new requests while the retiring generation finishes its existing runs. The relay adopts those runs without replay, preserving their schedule holds, cancellation and settled delivery receipts. After completion or rollback, the supervisor holds new turn admission until existing native work finishes and its successor worker is ready. Queued requests remain in the running relay and file transport; they do not block this supervisor reload. OS service process-group cleanup therefore cannot kill a retiring generation or be starved by newly admitted work. Plugin upgrades build while native work continues. Activation closes the existing registry admission lock, drains in-flight plugin command leases, and replaces the plugin under that lock. New native plugin calls wait for admission; an outdated revision fails before provider execution and must be rediscovered. No admitted provider operation is replayed. The shared broker and native sessions remain running. Older runtimes without the handoff contract retain the full native-work drain. Isolated agents migrate once after live native work finishes to a separate, unprivileged executor container. Pending requests remain queued for that executor; its private binding is copied into the container as root-owned, group-readable and not writable by the configured runtime user. Existing named volumes are attached as external volumes. It uses the same file transport; later relay/runtime upgrades preserve that executor generation. No native CLI or provider credentials move to the host. A plugin update also retains that drain if an operator replaced the broker’s single-file host binding (for example to approve new network pins), because that case still requires broker recreation.
 
 Compatible main upgrades also finish their advisory workspace/native-client
 backup before closing admission or stopping the relay. Large native histories do
@@ -63,6 +63,24 @@ restart and verify. Do not recreate the agent or its registry. Subsequent
 compatible upgrades are agent-owned. Noncanonical/multi-agent host configuration
 files require migration to one canonical deployment per service first.
 
+## Recovering an isolated-executor or supervisor admission defect
+
+The post-update reload requires a live relay status response taken after
+in-progress admissions finish. An unavailable or timed-out response keeps
+admission closed and leaves active executors running.
+
+If an older supervisor cannot start its isolated executor, or its post-update
+reload admits new work indefinitely, it can prevent its own correction from
+being installed. Use an exact,
+reviewed newer package's `src/updates/supervisor.mjs DEPLOYMENT_DIRECTORY` as the
+existing OS service's temporary entry point. First verify that no native run is
+active. Keep the same service user, environment, registry, deployment and queued
+requests; do not edit published package files or delete claims. This CLI runs the
+same supervisor and retains the active package binding until normal upgrade
+activation. Let existing queued jobs finish, then prepare and apply the corrected
+release through `ez updates`. Restore the retained bootstrap entry point after
+the active package and running services verify the corrected version.
+
 ## Package-manager preflight and repair
 
 Before installing candidate dependencies or stopping services, the supervisor

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@jc_stack/ez-agents",
-  "version": "0.1.0-beta.51",
+  "version": "0.1.0-beta.58",
   "private": false,
   "type": "module",
   "description": "A lightweight foundation for persistent business AI assistants using existing AI harnesses, workspaces and plugins.",

package/src/ai-cli.ts

@@ -2,15 +2,15 @@ import { parseArgs } from 'node:util'
 import { randomBytes } from 'node:crypto'
 import { join } from 'node:path'
 import { mkdir, readFile } from 'node:fs/promises'
-import { authProfileStatus, readModels, presetProvider, sameChoice, sameEngine, validateSelection, type AiPreset, type ModelChoice } from './ai.js'
-import { cliHome } from './auth-profile.js'
+import { authProfileStatus, readModels, presetLabel, presetProvider, resolveFallbacks, sameChoice, sameEngine, validateSelection, type AiPreset, type ModelChoice } from './ai.js'
+import { cliHome, provisionedHome } from './auth-profile.js'
 import { ControlStore } from './control-state.js'
 import { resolveIsolation } from './isolation.js'
 
-const usage='Usage: ezenciel-agents-ai list | select --cli <installed-cli> [--auth-profile <name>] [--provider <provider>] [--model <model>] [--effort <effort>] | default | profiles | profile add --cli codex|claude --auth-profile <name>'
-const {values,positionals}=parseArgs({allowPositionals:true,options:{help:{type:'boolean'},cli:{type:'string'},'auth-profile':{type:'string'},provider:{type:'string'},model:{type:'string'},effort:{type:'string'}}})
+const usage='Usage: ezenciel-agents-ai list | select --cli <installed-cli> [--auth-profile <name>] [--provider <provider>] [--model <model>] [--effort <effort>] | default | profiles | profile add --cli codex|claude --auth-profile <name> | fallbacks [--fallback cli=CLI[,auth-profile=NAME][,provider=ID][,model=MODEL][,effort=EFFORT] ... | --clear] | telemetry [on|off]'
+const {values,positionals}=parseArgs({allowPositionals:true,options:{help:{type:'boolean'},cli:{type:'string'},'auth-profile':{type:'string'},provider:{type:'string'},model:{type:'string'},effort:{type:'string'},fallback:{type:'string',multiple:true},clear:{type:'boolean'}}})
 if(values.help){
-  console.log(`${usage}\nChoose only values returned by list. Selection affects subsequent messages; default makes the selected AI the choice for new conversations. Queued work is unchanged.\nAuth profiles are separate owner-provisioned Codex/Claude logins with their own configuration and sessions; changing profile starts a new conversation. profile add creates an empty home and prints the native login command for the owner to run; profiles reports read-only login status without secrets.`)
+  console.log(`${usage}\nChoose only values returned by list. Selection affects subsequent messages; default makes the selected AI the choice for new conversations. Queued work is unchanged.\nAuth profiles are separate owner-provisioned Codex/Claude logins with their own configuration and sessions; changing profile starts a new conversation. profile add creates an empty home and prints the native login command for the owner to run; profiles reports read-only login status without secrets.\nfallbacks sets the ordered chain tried after the selected AI for chat and application turns, captured when work is queued. An entry for the selected AI's CLI without provider, model or effort keeps the turn's own model and effort. Core advances only when a setup is unavailable or rejected before work, or stopped at its usage limit (a Codex chat setup advances only when its CLI is unavailable); each other login resumes its own native session for that conversation. Without options it shows the chain.\ntelemetry on records per-attempt model, token, tool and error-category counts in run records (ezenciel-agents-schedule run RUN_ID); off by default; never prompts, tool arguments/results or costs.`)
   process.exit(0)
 }
 if (!process.env.EZ_CONTROL_DIR) throw new Error('Use this agent’s bound control directory')
@@ -29,6 +29,32 @@ else if(positionals[0]==='profile' && positionals[1]==='add'){
     : `CLAUDE_CONFIG_DIR=${home} claude auth login (or store a \`claude setup-token\` token in ${home}/oauth-token, mode 600)`
   console.log(JSON.stringify({cli:values.cli,authProfile,home,created:created!==undefined,ownerLogin:login}))
 }
+else if(positionals[0]==='fallbacks'){
+  const control=new ControlStore(controlDir,900000)
+  if(values.fallback && values.clear)throw new Error('Choose --fallback or --clear')
+  if(values.fallback){
+    const chain=[]
+    for(const [index,spec] of values.fallback.entries()){
+      const fields=Object.fromEntries(spec.split(',').map(part=>{const at=part.indexOf('=');if(at<1)throw new Error('Fallback uses cli=CLI[,auth-profile=NAME][,provider=ID][,model=MODEL][,effort=EFFORT]');return [part.slice(0,at),part.slice(at+1)]}))
+      if(!fields.cli || Object.keys(fields).some(key=>!['cli','auth-profile','provider','model','effort'].includes(key)))throw new Error('Fallback uses cli=CLI[,auth-profile=NAME][,provider=ID][,model=MODEL][,effort=EFFORT]')
+      const preset:AiPreset={id:`fallback-${index+1}`,name:'Fallback',cli:fields.cli,...(fields.provider ? {provider:fields.provider} : {}),...(fields['auth-profile'] ? {authProfile:fields['auth-profile']} : {}),...(fields.model ? {model:fields.model} : {}),...(fields.effort ? {effort:fields.effort} : {})}
+      if(preset.authProfile)await provisionedHome(controlDir,preset.cli,preset.authProfile)
+      await validateSelection(preset,await catalog())
+      chain.push({...preset,name:presetLabel(preset).slice(0,80)})
+    }
+    await control.setChatFallbacks(chain)
+  } else if(values.clear) await control.setChatFallbacks(undefined)
+  const state=await control.status()
+  const selected=state.ai?.presets.find(p=>p.id===state.ai?.selectedId)
+  console.log(JSON.stringify({fallbacks:state.ai?.fallbacks ?? [],...(selected ? {selected:presetLabel(selected),order:[selected,...resolveFallbacks(selected,state.ai?.fallbacks) ?? []].map(presetLabel)} : {}),
+    applies:'chat and application work queued afterwards; queued work keeps its captured chain'}))
+}
+else if(positionals[0]==='telemetry'){
+  const control=new ControlStore(controlDir,900000)
+  if(positionals[1]==='on' || positionals[1]==='off')await control.setRunTelemetry(positionals[1]==='on')
+  else if(positionals[1]!==undefined)throw new Error(usage)
+  console.log(JSON.stringify({runTelemetry:(await control.status()).runTelemetry===true ? 'on' : 'off',applies:'runs started afterwards'}))
+}
 else if(positionals[0]==='default'){
   const control=new ControlStore(controlDir,900000)
   const state=await control.status()

package/src/ai.ts

@@ -10,7 +10,8 @@ import { desktopCodexPath } from './desktop-bridge.js'
 import { AUTH_PROFILE_CLIS, claudeAuthEnvironment, cliHome, isAuthProfile, listAuthProfiles } from './auth-profile.js'
 
 export type AiPreset = { id: string; name: string; cli: string; provider?: string; authProfile?: string; model?: string; effort?: string }
-// Optional ordered fallbacks are tried only for a scheduled occurrence that never started work.
+// Ordered fallback setups (scheduled tasks, or the owner's chat chain captured
+// when work is queued) retain the native rejection/partial-work boundaries.
 export type ExecutionChoice = { sessionId: string; preset: AiPreset; fallbacks?: AiPreset[] }
 export type ModelChoice = { cli: string; provider?: string; authProfile?: string; model?: string; name: string; efforts: string[] }
 type Engine = { cli?: string; authProfile?: string }
@@ -35,6 +36,21 @@ export const isExecutionChoice = (v: unknown): v is ExecutionChoice => {
   return Boolean(c && /^[0-9a-f-]{36}$/i.test(c.sessionId) && isPreset(c.preset) &&
     (c.fallbacks === undefined || (Array.isArray(c.fallbacks) && c.fallbacks.length > 0 && c.fallbacks.every(isPreset))))
 }
+// The owner's chat fallback chain for one captured primary choice. A same-client
+// entry without provider, model or effort keeps the primary's settings, so other
+// logins preserve each conversation's or role's model and effort. Entries equal
+// to an earlier setup are dropped.
+export const resolveFallbacks = (primary: AiPreset, chain: AiPreset[] = []): AiPreset[] | undefined => {
+  const resolved: AiPreset[] = []
+  for (const entry of chain) {
+    const inherit = entry.cli === primary.cli && entry.provider === undefined && entry.model === undefined && entry.effort === undefined
+    const { provider, model, effort } = inherit ? primary : entry
+    const preset: AiPreset = { id: entry.id, name: entry.name, cli: entry.cli, ...(provider ? { provider } : {}),
+      ...(entry.authProfile ? { authProfile: entry.authProfile } : {}), ...(model ? { model } : {}), ...(effort ? { effort } : {}) }
+    if (![primary, ...resolved].some(other => sameChoice(other, preset))) resolved.push({ ...preset, name: presetLabel(preset).slice(0, 80) })
+  }
+  return resolved.length ? resolved : undefined
+}
 export const presetLabel = (p: AiPreset) => `${p.cli}${p.authProfile ? `@${p.authProfile}` : ''}${p.provider ? ` (${p.provider})` : ''} · ${p.model || 'client default'} · ${p.effort || 'default effort'}`
 // OpenCode encodes its provider in the native provider/model identifier.
 export const presetProvider = (p: AiPreset) => p.cli === 'opencode' ? p.model?.split('/')[0] : p.provider

package/src/codex-session.ts

@@ -6,8 +6,9 @@ import { fileURLToPath } from 'node:url'
 import path from 'node:path'
 import { terminateJob, validateCodexProvider, type CodexProviderBinding } from './executor.js'
 import { codexRejection, codexResetAt, startupLine } from './setup-fallback.js'
+import { telemetryObserver } from './run-telemetry.js'
 
-type Options = {workspace:string;controlDir:string;toolsHome?:string;sharedWorkspace?:string;additionalWorkspaces?:string[];model?:string;effort?:string;prompt:string;codexSandbox?:'external';codexProvider?:CodexProviderBinding}
+type Options = {workspace:string;controlDir:string;toolsHome?:string;sharedWorkspace?:string;additionalWorkspaces?:string[];model?:string;effort?:string;prompt:string;codexSandbox?:'external';codexProvider?:CodexProviderBinding;telemetry?:boolean}
 type Message = {id?:number;method?:string;params?:any;result?:any;error?:{message:string;code?:number}}
 
 // Keep Codex's native session alive. Codex itself starts goal continuation turns;
@@ -19,6 +20,15 @@ export async function runCodexSession(options:Options, io:{launch?:()=>ChildProc
   const child=io.launch?.() ?? spawn('codex',['app-server','--stdio','--disable','memories','--enable','skip_host_skill_discovery'],{cwd:options.workspace,env:process.env,stdio:['pipe','pipe','pipe']})
   const emit=io.emit ?? (line=>process.stdout.write(line+'\n'))
   let id=0,threadId:string|undefined,activeTurn:string|undefined,finished=false,sawTurn=false,hadGoal=false,workBegan=false,resetAt:string|undefined
+  // Owner-enabled telemetry: app-server items and usage in the exec event shape.
+  const telemetry=options.telemetry ? telemetryObserver() : undefined
+  let reported=''
+  const observe=(event?:object)=>{
+    if(!telemetry)return
+    if(event)telemetry.write(JSON.stringify(event)+'\n')
+    const line=telemetry.summary();if(line!==reported){reported=line;emit(line)}
+  }
+  const tools:Record<string,string>={commandExecution:'command_execution',fileChange:'file_change',webSearch:'web_search',mcpToolCall:'mcp_tool_call'}
   let resolveDone!:(code:number)=>void
   const done=new Promise<number>(resolve=>{resolveDone=resolve})
   const pending=new Map<number,{resolve:(value:any)=>void;reject:(error:Error)=>void;timer:ReturnType<typeof setTimeout>}>()
@@ -33,6 +43,7 @@ export async function runCodexSession(options:Options, io:{launch?:()=>ChildProc
   const settled=(goal:any)=>{
     if(goal)hadGoal=true
     if(activeTurn || goal?.status==='active')return
+    if(goal?.status==='usageLimited')emit(startupLine({workBegan,rejection:{category:'quota',...(resetAt?{resetAt}:{})}}))
     if(goal && goal.status!=='complete'){console.error(`Native goal stopped: ${goal.status}`);finish(1);return}
     if(!goal && hadGoal){fail(new Error('Native goal disappeared without verified completion'));return}
     if(!sawTurn)return
@@ -64,6 +75,11 @@ export async function runCodexSession(options:Options, io:{launch?:()=>ChildProc
     if(message.method==='account/rateLimits/updated')resetAt=codexResetAt(message.params?.rateLimits) ?? resetAt
     if(message.params?.threadId!==threadId)return
     if(message.method==='turn/started'){activeTurn=message.params.turn.id;sawTurn=true}
+    const item=message.params?.item
+    if((message.method==='item/started'||message.method==='item/completed') && tools[item?.type])
+      observe({type:message.method==='item/started'?'item.started':'item.completed',item:{id:item.id,type:tools[item.type],server:item.server,tool:item.tool,status:['failed','declined'].includes(item.status)?'failed':'completed'}})
+    if(message.method==='thread/tokenUsage/updated'){const total=message.params.tokenUsage?.total
+      observe({type:'turn.completed',usage:{input_tokens:total?.inputTokens,cached_input_tokens:total?.cachedInputTokens,output_tokens:total?.outputTokens}})}
     // Any item beyond the echoed input means the provider accepted the turn.
     if(message.method==='item/started' && message.params.item?.type!=='userMessage' && !workBegan){workBegan=true;emit(startupLine({workBegan}))}
     if(message.method==='thread/goal/updated')settled(message.params.goal)
@@ -73,7 +89,9 @@ export async function runCodexSession(options:Options, io:{launch?:()=>ChildProc
       if(message.params.turn.status!=='completed'){
         if(message.params.turn.status==='interrupted')finish(130)
         else {
-          const category=workBegan ? undefined : codexRejection(message.params.turn.error?.codexErrorInfo)
+          observe({type:'turn.failed'})
+          const rejected=codexRejection(message.params.turn.error?.codexErrorInfo)
+          const category=!workBegan || rejected==='quota' ? rejected : undefined
           if(category)emit(startupLine({workBegan,rejection:{category,...(category==='quota' && resetAt ? {resetAt} : {})}}))
           fail(new Error(redactFailure(typeof message.params.turn.error?.message==='string'
           ? message.params.turn.error.message : 'Native turn failed without error details')))
@@ -88,6 +106,8 @@ export async function runCodexSession(options:Options, io:{launch?:()=>ChildProc
   try{
     await request('initialize',{clientInfo:{name:'ezenciel-agents',version:'1'},capabilities:{experimentalApi:true}})
     send({method:'initialized',params:{}})
+    // Read native quota metadata even when a rejected turn emits no update.
+    await request('account/rateLimits/read',{}).then(limits=>{resetAt=codexResetAt(limits.rateLimits)}).catch(()=>{})
     const result=await request('thread/start',{
       cwd:options.workspace,approvalPolicy:'never',sandbox:options.codexSandbox === 'external' ? 'danger-full-access' : 'workspace-write',model:options.model,
       config:{project_root_markers:['AGENTS.md','.git'],'sandbox_workspace_write.writable_roots':[options.controlDir,...(options.toolsHome?[options.toolsHome]:[]),...(options.sharedWorkspace?[options.sharedWorkspace]:[]),...(options.additionalWorkspaces ?? [])],
@@ -96,6 +116,7 @@ export async function runCodexSession(options:Options, io:{launch?:()=>ChildProc
     })
     threadId=result.thread?.id
     if(!threadId)throw new Error('Codex did not return a native thread ID')
+    telemetry?.model(result.model)
     emit(JSON.stringify({type:'thread.started',thread_id:threadId}))
     await request('turn/start',{threadId,input:[{type:'text',text:options.prompt}],model:options.model,effort:options.effort,
       ...(options.codexSandbox === 'external' ? {sandboxPolicy:{type:'externalSandbox',networkAccess:'enabled'}} : {})})