@staix/agent-hub 0.12.28 → 0.12.30
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +24 -0
- package/README.md +79 -219
- package/docs/agent-notes/README.md +12 -0
- package/docs/agent-notes/adapters.md +1 -1
- package/docs/agent-notes/bus.md +3 -1
- package/docs/agent-notes/daemon.md +13 -0
- package/docs/agent-notes/local-worker.md +2 -2
- package/docs/agent-notes/tasks.md +9 -0
- package/docs/agent-notes/tests.md +1 -1
- package/docs/memory.md +70 -152
- package/docs/operations.md +237 -14
- package/docs/plans/2026-10-16-amem-failure-registry.md +101 -0
- package/docs/plans/2026-10-16-amem-p0-baseline.md +110 -0
- package/docs/plans/2026-10-16-amem-p2-read-tools.md +28 -0
- package/docs/plans/amem-cases.json +482 -0
- package/docs/quickstart.md +9 -2
- package/docs/security.md +39 -2
- package/docs/smoke.md +65 -0
- package/docs/specs/2026-09-19-agent-hub-design.md +113 -5
- package/docs/verification/2026-10-16-0.12.29.md +25 -0
- package/docs/verification/2026-10-16-273-live-verification.md +150 -0
- package/docs/verification/2026-10-17-0.12.30.md +70 -0
- package/docs/verified.json +13 -13
- package/package.json +5 -2
- package/plugins/agent-hub/.claude-plugin/plugin.json +1 -1
- package/plugins/agent-hub/server.js +64 -44
- package/src/cli/console-state.ts +92 -20
- package/src/cli/console.ts +25 -6
- package/src/cli/help.ts +4 -2
- package/src/cli/identity-audit.ts +2 -2
- package/src/cli/identity.ts +15 -1
- package/src/cli/main.js +2 -2
- package/src/cli/main.ts +78 -39
- package/src/cli/memory.ts +87 -187
- package/src/cli/output.ts +6 -0
- package/src/cli/recovery-package.ts +59 -10
- package/src/cli/upgrade-runtime.ts +60 -18
- package/src/cli/upgrade.ts +8 -0
- package/src/hub/board.ts +2 -1
- package/src/hub/bus.ts +33 -12
- package/src/hub/daemon.ts +141 -31
- package/src/hub/envelope.ts +30 -0
- package/src/hub/events.ts +1 -1
- package/src/hub/git.ts +2 -38
- package/src/hub/hub-tools.ts +5 -1
- package/src/hub/lifecycle.ts +2 -2
- package/src/hub/manager.ts +15 -11
- package/src/hub/memory-candidates.ts +103 -0
- package/src/hub/memory-fail.ts +25 -0
- package/src/hub/memory-observations.ts +47 -0
- package/src/hub/memory-plan-guard.ts +33 -0
- package/src/hub/memory-policy.ts +42 -0
- package/src/hub/memory-profile-check.ts +170 -0
- package/src/hub/memory-profile.ts +192 -0
- package/src/hub/memory-staleness.ts +55 -0
- package/src/hub/path-policy.ts +8 -0
- package/src/hub/project.ts +4 -37
- package/src/hub/read-tools.ts +127 -0
- package/src/hub/tasks.ts +201 -41
- package/src/hub/ui.ts +10 -7
- package/src/local/deny.ts +4 -17
- package/src/memory/brief.ts +8 -5
- package/src/memory/capture.ts +2 -1
- package/src/memory/recall.ts +50 -110
- package/src/pi/process-signature.ts +2 -42
- package/src/memory/import.ts +0 -90
- package/src/memory/store.ts +0 -805
- package/src/memory/sync.ts +0 -145
package/docs/operations.md
CHANGED
|
@@ -1,8 +1,46 @@
|
|
|
1
1
|
# Operations guide
|
|
2
2
|
|
|
3
|
-
This guide describes ahub 0.12.
|
|
3
|
+
This guide describes ahub 0.12.30 and control protocol 16. Live verification
|
|
4
4
|
results and remaining prerequisites are recorded separately in [the smoke ledger](smoke.md).
|
|
5
5
|
|
|
6
|
+
## Agent-operated live verification
|
|
7
|
+
|
|
8
|
+
When the user explicitly authorizes a live test, its agent may run console
|
|
9
|
+
commands using `ahub --live-test [--project <path|id>] <command> [args...]`.
|
|
10
|
+
For example, `ahub --live-test --project <scratch-path> ui --no-open` starts the
|
|
11
|
+
real dashboard listener and prints its one-time link without opening a browser.
|
|
12
|
+
`ahub --project <scratch-path> --live-test reset --all --yes` exercises reset on
|
|
13
|
+
that selected disposable project. The global options precede the command and
|
|
14
|
+
may appear in either order; a flag after the command is not delegation.
|
|
15
|
+
|
|
16
|
+
This grants console authority for operational console-class and peer-lifecycle conductor-class commands, including destructive actions, for one
|
|
17
|
+
invocation and its ahub continuations. Use only the user-authorized targets and
|
|
18
|
+
commands; destructive live checks belong in disposable projects. Existing
|
|
19
|
+
project discovery still applies: initialize the scratch folder as its own Git
|
|
20
|
+
repository before `ahub init`, then confirm `status --json` reports that exact
|
|
21
|
+
`cwd` before any destructive check.
|
|
22
|
+
Existing confirmations, process ownership checks, native identity fences and recovery
|
|
23
|
+
locks still apply. Do not remove agent markers. The initiating peer and command
|
|
24
|
+
label are audited before effects; an unavailable audit blocks delegation. A new
|
|
25
|
+
unflagged invocation has ordinary peer authority again. Native tools launched
|
|
26
|
+
by the command keep their own peer identity and receive no delegation flag.
|
|
27
|
+
Allowed-class commands keep peer identity and permissions even with the flag:
|
|
28
|
+
`--live-test say` sends as the peer. Peer starts, pause and resume may use
|
|
29
|
+
delegated console authority. Flagged task assign/escalate refuse; omit the flag
|
|
30
|
+
to use the ordinary actor-preserving conductor path when the peer has that role.
|
|
31
|
+
Person verdicts and user-authored workflow writes are not delegated. Dashboard
|
|
32
|
+
sessions issued through the flag retain operational controls but refuse message
|
|
33
|
+
and task/planning content writes. Use native peer tools for agent-authored work.
|
|
34
|
+
|
|
35
|
+
Use a CLI/coordinator containing this feature throughout the experiment. Older
|
|
36
|
+
daemons do not understand the new delegated audit outcome; run the candidate
|
|
37
|
+
daemon to capture its delegated event and console notice. This policy does not
|
|
38
|
+
change external computer-use restrictions. Recovery staging refuses old targets
|
|
39
|
+
without live-test support before source effects. Delegated headless recovery restores through the ordinary target CLI, carrying
|
|
40
|
+
its operationId on the delegated console connection; existing native ownership
|
|
41
|
+
and session-identity checks remain. Record actual observations in the
|
|
42
|
+
smoke ledger; the presence of this option proves no outstanding #273 live leg.
|
|
43
|
+
|
|
6
44
|
## Command output
|
|
7
45
|
|
|
8
46
|
`ahub status`, `board`, `budget`, `doctor`, `projects`, `queue list`, `turns` and `report` print shared console tables and labelled details.
|
|
@@ -138,6 +176,48 @@ decisions and a never-ask permission mode require confirmation; delivery resolut
|
|
|
138
176
|
status state counts and never fetches the board. Leaving restores
|
|
139
177
|
the terminal and returning from panels replays the bounded stream buffer.
|
|
140
178
|
|
|
179
|
+
Tasks and Queue selections follow the selected id across refreshes. If that row
|
|
180
|
+
disappears, no replacement row inherits the action. Task actions also bind the
|
|
181
|
+
row's `flowRevision` (its history length); queue actions bind the delivery
|
|
182
|
+
revision. The id and revision in a row-started command cannot be edited. Peer,
|
|
183
|
+
reason and the final role/action remain editable, and the confirmation names the
|
|
184
|
+
validated result. An older hub without `flowRevision` refuses revision-bound task
|
|
185
|
+
row actions instead of silently running them unchecked.
|
|
186
|
+
|
|
187
|
+
In Tasks, `a` assigns the owner in ordinary open states and the reviewer in
|
|
188
|
+
`in_review`; a waiting task says when assignment is a reservation. `u` unassigns
|
|
189
|
+
the corresponding role, and `r` reviews only in `in_review`. In Queue, `r`
|
|
190
|
+
prefills `discard` for a queued row and `retry` for a needs_review row, with the
|
|
191
|
+
action still editable before confirmation. `completed` is never the default.
|
|
192
|
+
Every resolution needs a reason. Queue `discard` removes queued originals or
|
|
193
|
+
settles a needs_review receipt; it never cancels a task or native turn.
|
|
194
|
+
|
|
195
|
+
`ahub task unassign <id> [--role owner|reviewer] [--revision N]` is the terminal
|
|
196
|
+
equivalent. Owner removal is allowed in `proposed`, `in_progress` and
|
|
197
|
+
`changes_requested`, returns active work to `proposed`, and clears any owner
|
|
198
|
+
reservation. It is refused in `in_review`, `approved` and `backlog`. Reviewer
|
|
199
|
+
removal retains the state, including `in_review`, and refuses to remove the human
|
|
200
|
+
reviewer of PII work. No replacement is routed and no native process or external
|
|
201
|
+
effect is rolled back. A person-unassigned task stays out of automatic owner
|
|
202
|
+
offers across dependency release, planner-pull processing, release timers and
|
|
203
|
+
restarts until an explicit assignment or reservation. The later unassign also
|
|
204
|
+
fences planner undo, so undo and re-pull cannot restore the removed reservation.
|
|
205
|
+
Former queued assignment or review requests are withdrawn without resolving
|
|
206
|
+
uncertain deliveries, and the former agent loses current authority. A withdrawal
|
|
207
|
+
notice uses delivery evidence or that peer's own accept/self-claim after the
|
|
208
|
+
latest assignment; another owner's in-progress state or an earlier acceptance
|
|
209
|
+
does not count. Withdrawing an unread queue alone sends none. If a later person
|
|
210
|
+
assignment supersedes an escalation or offline relay, the stale success result
|
|
211
|
+
and decision note are suppressed, but the former owner is still told the current
|
|
212
|
+
owner and to stop working.
|
|
213
|
+
|
|
214
|
+
`ahub task assign <id> <peer> [--role owner|reviewer] [--revision N]` uses a
|
|
215
|
+
checked console operation when a revision is present. Without it, existing CLI
|
|
216
|
+
behavior is unchanged. `ahub queue resolve ... [--revision N]` uses the supplied
|
|
217
|
+
revision instead of a fresh show. Revision mismatches and superseded post-await
|
|
218
|
+
publications fail without a board or delivery side effect; a reviewer-only change
|
|
219
|
+
keeps a valid owner assignment but refreshes its reviewer content.
|
|
220
|
+
|
|
141
221
|
The Tasks panel summarizes the whole public board with an approved/all fraction,
|
|
142
222
|
a 20-cell ASCII bar and state counts. Proposed tasks with unapproved or missing
|
|
143
223
|
`after` dependencies count as waiting; PII stubs count normally. Each task has a
|
|
@@ -350,9 +430,136 @@ The Stop hook acknowledgement is not a counted completion. The daemon checks the
|
|
|
350
430
|
native transcript after the hook can return; missing or changed-session evidence
|
|
351
431
|
stays unknown.
|
|
352
432
|
|
|
433
|
+
## Pi with DGX and MLX
|
|
434
|
+
|
|
435
|
+
Pi is optional. Install Pi separately and set `pi.enabled` to `true` in
|
|
436
|
+
`.agenthub/config.json`. Set `pi.auto_start` to `true` to start it with the hub.
|
|
437
|
+
Pi starts in its TUI by default; `ahub pi --headless`, or the start mode
|
|
438
|
+
setting (`ahub settings set peers.pi.start_mode headless`), asks for the
|
|
439
|
+
background peer instead. The project uses its existing OmniRoute connection for DGX;
|
|
440
|
+
`pi.dgx_coding` and `pi.dgx_fast` select the gateway's configured model IDs.
|
|
441
|
+
|
|
442
|
+
```bash
|
|
443
|
+
ahub models setup # prepare bounded Qwen3.5 4B MLX in an existing Ollama service
|
|
444
|
+
ahub models start # verify loopback Ollama/model readiness (no daemon spawn)
|
|
445
|
+
ahub models status
|
|
446
|
+
ahub pi --headless --backend auto
|
|
447
|
+
ahub say @pi "Read the project and summarize the next small implementation task"
|
|
448
|
+
ahub pi # its TUI in this terminal; an idle headless session moves into it
|
|
449
|
+
```
|
|
450
|
+
|
|
451
|
+
The default routing template assigns implementation, bulk edits and tests to
|
|
452
|
+
DGX, and summaries/triage to MLX. Pi and the existing local worker take priority
|
|
453
|
+
before cloud peers; planning and final review stay with Claude/Codex. Existing
|
|
454
|
+
projects keep their routing file, so add `pi` to the relevant `peers` lists and
|
|
455
|
+
set `pi_backend = "dgx"` or `"mlx"` for each class. PII tasks remain local-worker
|
|
456
|
+
only. `--backend dgx` or `--backend mlx` explicitly pins a Pi session's backend.
|
|
457
|
+
|
|
458
|
+
Pi connects to an authenticated relay with model aliases `dgx/coding`,
|
|
459
|
+
`dgx/fast`, and `mlx/fast`; upstream gateway credentials stay in the hub.
|
|
460
|
+
Its managed tools use the existing path guards, shell sandbox and terminal
|
|
461
|
+
approval flow. Keep `ahub console` open to approve write/edit/shell operations:
|
|
462
|
+
a request nobody answers expires, and two in a row end Pi's turn.
|
|
463
|
+
Tool-call receipts survive daemon restart; an interrupted operation with an
|
|
464
|
+
unknown outcome must be reconciled before repeating it. Cancelling a Pi turn
|
|
465
|
+
does not automatically hand its task to a cloud peer.
|
|
466
|
+
|
|
467
|
+
The `mlx/fast` alias now defaults to an externally managed Ollama MLX backend.
|
|
468
|
+
Install and start Ollama separately, with `OLLAMA_NO_CLOUD=1`,
|
|
469
|
+
`OLLAMA_NUM_PARALLEL=1`, `OLLAMA_MAX_LOADED_MODELS=1`, and a finite
|
|
470
|
+
`OLLAMA_KEEP_ALIVE` (five minutes is the tested setting). `models setup`
|
|
471
|
+
prepares `agenthub-fast-mlx:4b-8k` from `qwen3.5:4b-mlx`, with an 8K total
|
|
472
|
+
context and a 2K output cap. The relay budgets input separately at 6,000
|
|
473
|
+
estimated tokens and serializes local requests across project hubs.
|
|
474
|
+
|
|
475
|
+
`models status` distinguishes model availability from memory residency.
|
|
476
|
+
`models stop` refuses to unload an externally managed Ollama model: other
|
|
477
|
+
clients may be using it. Ollama owns idle eviction; stopping a hub never
|
|
478
|
+
kills Ollama. The OpenAI compatibility endpoint does not enforce a request
|
|
479
|
+
`keep_alive` field, so configure the server and verify its expiry readback.
|
|
480
|
+
The relay can fall back to DGX before streaming starts as before. See the
|
|
481
|
+
[Ollama migration procedure](#ollama-mlx-migration-issue-51)
|
|
482
|
+
for existing Python runtime/configuration cleanup and rollback.
|
|
483
|
+
|
|
484
|
+
A project has one managed Pi session owner. Mode changes require the agent to
|
|
485
|
+
settle and preserve the session file. Before the first generation, a live source
|
|
486
|
+
verified to be empty can instead retain its exact session ID without a file.
|
|
487
|
+
Missing history from a nonempty session blocks handover. Protocol 9 adds Pi
|
|
488
|
+
recovery metadata. Protocol 10, released in 0.7.0, adds durable delivery
|
|
489
|
+
receipts and conservative queue recovery; it is verified live (see the
|
|
490
|
+
[smoke checklist](smoke.md)).
|
|
491
|
+
Older protocol hubs still require their matching CLI and an attended bootstrap.
|
|
492
|
+
|
|
493
|
+
## Local dashboard
|
|
494
|
+
|
|
495
|
+
With a running daemon, `ahub ui` opens a local dashboard for messages, peer states
|
|
496
|
+
and queues, the task board, budget windows, pending approvals and this machine's
|
|
497
|
+
benchmark runs. Use `ahub ui --no-open` to print a single-use link when the
|
|
498
|
+
browser cannot be opened.
|
|
499
|
+
Open the link within 60 seconds; the browser session lasts one hour.
|
|
500
|
+
|
|
501
|
+
The dashboard can send console messages, propose and assign tasks, pause or resume
|
|
502
|
+
peers, answer approvals, start a peer in its start mode and show the settings.
|
|
503
|
+
An ordinary session changes the safe switches and lowers a mode to `ask`; the
|
|
504
|
+
rest needs `ahub ui --settings` or `ahub settings` in a terminal. See
|
|
505
|
+
[Settings](#settings-from-the-dashboard-and-the-terminal) and
|
|
506
|
+
[Start modes](#start-modes-tui-by-default-headless-as-opt-in).
|
|
507
|
+
Budget pauses still require the terminal override.
|
|
508
|
+
Private envelopes and PII tasks stay redacted. Local-worker and Pi tool details and
|
|
509
|
+
allow decisions stay in `ahub tail` / `ahub permit`; the page can deny them.
|
|
510
|
+
No dashboard port is opened until requested. See [the session security model](security.md#local-dashboard-sessions-issue-6).
|
|
511
|
+
|
|
512
|
+
## Multiple projects and worktrees
|
|
513
|
+
|
|
514
|
+
Version 0.4.0 adds one independent hub per repository or Git worktree.
|
|
515
|
+
Subdirectory commands resolve to that project; symlink aliases resolve to the same
|
|
516
|
+
hub. Explicit selection works from any directory:
|
|
517
|
+
|
|
518
|
+
```bash
|
|
519
|
+
ahub --project /path/to/project-a init
|
|
520
|
+
ahub --project /path/to/project-b init
|
|
521
|
+
ahub --project /path/to/project-a up
|
|
522
|
+
ahub --project /path/to/project-b up
|
|
523
|
+
ahub projects
|
|
524
|
+
ahub status --all
|
|
525
|
+
ahub --project /path/to/project-a codex
|
|
526
|
+
ahub ui --all
|
|
527
|
+
```
|
|
528
|
+
|
|
529
|
+
The unified dashboard lists registered roots and opens their boards, streams,
|
|
530
|
+
messages, approvals and budgets. It can start hubs and stop a selected hub after
|
|
531
|
+
confirmation. Its Start control opens a peer's TUI only through a terminal provider
|
|
532
|
+
(`terminal.open` or Orca); without one, launch Claude/Codex TUIs from a terminal. The manager stays available
|
|
533
|
+
when a project stops; `ahub ui --all --stop` stops only the manager. Existing
|
|
534
|
+
`ahub ui` still opens the current project's dashboard.
|
|
535
|
+
|
|
536
|
+
`ahub projects --json` returns machine-readable live status. Use a listed ID or a
|
|
537
|
+
path with `--project`; the selector must precede the command. `ahub projects remove
|
|
538
|
+
<id>` forgets a stopped registration without deleting project data. A moved root
|
|
539
|
+
gets a new registration. Missing roots remain visible until removed.
|
|
540
|
+
|
|
541
|
+
Runtime state stays in each project's `.agenthub/state/`. The machine registry is
|
|
542
|
+
`~/.agenthub/registry.db`; existing `ports.json` allocations are imported once.
|
|
543
|
+
`AGENTHUB_HOME` selects an isolated machine registry (also used by tests). An
|
|
544
|
+
inherited state directory from a different project cannot redirect a new CLI
|
|
545
|
+
invocation. Explicit custom state requires a matching project marker or the
|
|
546
|
+
initialization pair `AGENTHUB_PROJECT_DIR` / `AGENTHUB_STATE_DIR`.
|
|
547
|
+
|
|
548
|
+
Manager-started hubs use attended mode and project configuration, including
|
|
549
|
+
credential-file references. They do not inherit the launching project's gateway
|
|
550
|
+
URL/key overrides. Native memory remains shared according to claude-mem aliases;
|
|
551
|
+
worktrees recall their parent and composite identity. A shared-alias notice does
|
|
552
|
+
not mean that task boards or agent sessions are shared. Provider quota readings
|
|
553
|
+
may describe the same account and should not be added across projects.
|
|
554
|
+
|
|
555
|
+
Before upgrading an older hub, inspect the plan with the matching CLI.
|
|
556
|
+
Incompatible or unverified processes are shown explicitly and are never killed by
|
|
557
|
+
PID-name matching. See the operations guide and the
|
|
558
|
+
[multi-project specification](specs/2026-09-19-multi-project-design.md).
|
|
559
|
+
|
|
353
560
|
## Install and start
|
|
354
561
|
|
|
355
|
-
Use Bun 1.
|
|
562
|
+
Use Bun 1.4.2 or newer. Install the released package and install its Claude
|
|
356
563
|
channel plugin:
|
|
357
564
|
|
|
358
565
|
```bash
|
|
@@ -434,8 +641,9 @@ does not list), needs its directory in `local.read_allow`, for example
|
|
|
434
641
|
hub's tools: list a peer and it keeps only the capabilities named, from
|
|
435
642
|
`propose` (`hub_task_propose`), `assign` (proposing with another peer as owner),
|
|
436
643
|
`remember` (the `hub_remember` tool; the notes the hub itself keeps of done
|
|
437
|
-
summaries and review verdicts are not gated)
|
|
438
|
-
|
|
644
|
+
summaries and review verdicts are not gated), `memory_correct` (propose a local correction candidate for person review), `memory_read`
|
|
645
|
+
(`hub_memory_search`, `hub_memory_show`, `hub_memory_graph`) and `important`
|
|
646
|
+
(`[IMPORTANT]` messages). For example `"capabilities": { "local": ["propose", "remember"] }`.
|
|
439
647
|
A peer that is not listed keeps all of them, a listed peer whose value is not a
|
|
440
648
|
list gets none, and an unknown capability name grants nothing; `hub.log` says
|
|
441
649
|
so for each, and logs every refusal (`capabilities:`). A refused tool call says
|
|
@@ -485,8 +693,7 @@ resolves an uncertain delivery automatically. The same rule holds on every path
|
|
|
485
693
|
assigns a task (#297): once the move is saved on the board, a failed delivery prints
|
|
486
694
|
exactly one console notice ("the move is saved... delivery is not confirmed"), printed
|
|
487
695
|
where the error is thrown, and the caller (a peer in its turn, or the CLI) still gets
|
|
488
|
-
its error. Two exceptions, by construction: a move superseded while the assignment was being prepared, meaning a later history entry that moves or settles the task (another assignment, an unassign, the owner's accept, done, approval, close; the hub's quiet bookkeeping such as a late check result or a reviewer change does not supersede), is
|
|
489
|
-
the caller's error as before, with no notice naming either owner; and a claim is its own hand-over, so its
|
|
696
|
+
its error. Two exceptions, by construction: a move superseded while the assignment was being prepared, meaning a later history entry that moves or settles the task (another assignment, an unassign, the owner's accept, done, approval, close; the hub's quiet bookkeeping such as a late check result or a reviewer change does not supersede), returns the current task without a failed tool call and with no notice naming either owner; and a claim is its own hand-over, so its
|
|
490
697
|
accepted write and line stay outside the delivery rule (its overlap and routing announcements to earlier
|
|
491
698
|
owners are deliveries and share it). The repair is always
|
|
492
699
|
to look for a queued or needs_review delivery with
|
|
@@ -697,7 +904,7 @@ ahub task label 12 reverted # a later verdict (ok, regressed, reverted, i
|
|
|
697
904
|
|
|
698
905
|
## Hub memory
|
|
699
906
|
|
|
700
|
-
The hub
|
|
907
|
+
The hub opens the selected memory profile under `memoryHome()` (default `~/.agenthub/memory/`) beside claude-mem ([how it works](memory.md)): notes saved
|
|
701
908
|
against an open task are short-term; when the task is approved its decisions, contracts and fails become long-term.
|
|
702
909
|
Task briefs, session-start recall and `ahub ask` read it first, and the dashboard has a Memory section.
|
|
703
910
|
|
|
@@ -710,6 +917,20 @@ ahub memory forget 1a2b3c4d # a person's; also promote, demote, sup
|
|
|
710
917
|
|
|
711
918
|
`"memory": { "native": false }` in `.agenthub/config.json` turns it off for a project.
|
|
712
919
|
|
|
920
|
+
The exact-pinned `@staix/agent-mem` 0.3.0 resolves `AMEM_HOME`, then `$AGENTHUB_HOME/memory`, then `~/.agenthub/memory`. An absent `M/profile.json` selects shared v1; an explicit generation selects `M/generations/G` with the package's readonly legacy bridge. All daemon memory consumers and the hub memory/goal CLI use that selected profile. Invalid metadata or a missing selected generation makes native memory unavailable while the bus and board continue; there is no fallback to legacy writes. Inspect the daemon's memory binding/availability before interpreting missing memory as an empty store.
|
|
921
|
+
|
|
922
|
+
The daemon checks its selector at hub operation entry and before candidate/counter writes. It does not hot-switch handles or provide an atomic fence against selection changes during an in-flight core call. Quiesce consumers and close their handles before migration or selection changes; reopen through the supported session-preserving lifecycle afterward. Candidate/counter storage remains `M/candidates.db`, independent of `G`; `amem review` on a generation is still refused. Every running hub using this home needs an authenticated receipt: a matching ready binding, or a generation-capable unbound receipt (`disabled` when native memory is off, `unavailable` when opening failed) proving it holds no memory handle. An unbound daemon and zero running hubs are quiesced for memory; a stale bound handle, unreadable registry or unknown runtime is not ready. Existing JSON memory/goal results remain unchanged; the additive binding receipt belongs to daemon status, profile-check and the decay object. This source integration does not authorize installing a new runtime, migrating real memory, restarting sessions or claiming real-machine sync/capture receipts. Migration/deactivation and their limits belong to [agent-mem](https://github.com/STAIxBWLB/agent-mem); see [the hub profile contract](memory.md#selected-memory-profile).
|
|
923
|
+
|
|
924
|
+
Run `ahub memory profile-check [--json]` from the person's terminal for the read-only preflight. It identifies live registry owners by signed process identity and authenticated daemon replies, then reports their loaded memory package and bound home/profile. The receipt names the current `AGENTHUB_HOME` registry scope; zero results establish no live daemon in that inventory, not system-wide process absence. Inventory and check separate custom registries too when they share `AMEM_HOME`. A missing or unreadable runtime receipt never counts as a different home or permission to migrate.
|
|
925
|
+
|
|
926
|
+
Before real migration, first upgrade every hub sharing the home to 0.12.30. Obtain positive ready or unbound/quiesced receipts in every relevant registry; then quiesce consumers and close bound handles before changing the selector. A matching ready handle proves compatibility but is still an open handle. Verify synthetic cross-writes before the separately approved real migration, then reopen through the supported session-preserving lifecycle afterward. Merely installing this release does not authorize migration or provide live capture/device-sync evidence.
|
|
927
|
+
|
|
928
|
+
Managed agents read through `hub_memory_search`, `hub_memory_show` and `hub_memory_graph`, with the server's project identity, the `memory_read` capability and current privacy screening. They cannot supply another project; private turns are refused. The tools bound UTF-8 input/output, screen before counts and truncation, and exclude their arguments/results from hub-managed capture. `hub_memory_correct` has its own `memory_correct` capability and records a screened correction candidate, not a note change. See [the complete memory contract](memory.md) for caps and refusal semantics.
|
|
929
|
+
|
|
930
|
+
Review candidates and rendered/delivered/cited counters stay in machine-local `M/candidates.db`, outside sync, import, recall, briefs, ask and graph evidence. Public changes-requested reviews with unmet items may propose a fail candidate; missing cause/retry details stay unknown. Shared-v1 acceptance is a person's `amem review` action with its TTY/policy checks; selected generations still refuse that command. Read-time `(stale?)` hints and plan-fail advice never change stored rows or task authority. Structured fail notes are validated against 4000 UTF-8 bytes before screening or writes, with no silent truncation.
|
|
931
|
+
|
|
932
|
+
Recall presentation omits generated task-approval outcome notes and shows only nonempty reasons from generated changes-requested notes; an ordinary decision that merely resembles a header stays visible. Contracts and fails remain eligible. Capped recall lines preserve memory headers and end at word boundaries, and one delivery deduplicates ids across restored/current prefaces and shared note lines. Stored notes and brief evidence are unchanged; platform recall retains its budget split.
|
|
933
|
+
|
|
713
934
|
Between your machines, through a private git remote, with long-term tasks that span them:
|
|
714
935
|
|
|
715
936
|
```bash
|
|
@@ -1621,21 +1842,21 @@ section is what they run.
|
|
|
1621
1842
|
|
|
1622
1843
|
Upgrade running projects with the target release's own coordinator. It accepts
|
|
1623
1844
|
a running source on control protocol 9 (0.6.x), 10 (0.7.0 through 0.12.0),
|
|
1624
|
-
11 (0.12.1 and 0.12.2), 12 (0.12.3), 13 (0.12.4 through 0.12.15), 14 (0.12.16), 15 (0.12.17 through 0.12.19) or 16 (0.12.20 through 0.12.
|
|
1845
|
+
11 (0.12.1 and 0.12.2), 12 (0.12.3), 13 (0.12.4 through 0.12.15), 14 (0.12.16), 15 (0.12.17 through 0.12.19) or 16 (0.12.20 through 0.12.30), and only
|
|
1625
1846
|
a target on its own protocol, so the target's coordinator fits every supported
|
|
1626
1847
|
source and carries every recovery fix released up to it. Protocol 8 and older
|
|
1627
1848
|
(0.5.x and earlier) are refused as `manual-bootstrap-required`. Run from the
|
|
1628
1849
|
project directory, without replacing the global CLI first:
|
|
1629
1850
|
|
|
1630
1851
|
```bash
|
|
1631
|
-
bunx --package @staix/agent-hub@0.12.
|
|
1632
|
-
bunx --package @staix/agent-hub@0.12.
|
|
1852
|
+
bunx --package @staix/agent-hub@0.12.30 ahub upgrade --to 0.12.30 --dry-run
|
|
1853
|
+
bunx --package @staix/agent-hub@0.12.30 ahub upgrade --to 0.12.30 --yes
|
|
1633
1854
|
```
|
|
1634
1855
|
|
|
1635
1856
|
| Running now | Coordinator to use |
|
|
1636
1857
|
| --- | --- |
|
|
1637
1858
|
| 0.6.x (protocol 9) | the target's, through `bunx` as above |
|
|
1638
|
-
| 0.7.0 through 0.12.0 (protocol 10), 0.12.1 and 0.12.2 (protocol 11), 0.12.3 (protocol 12), 0.12.4 through 0.12.15 (protocol 13), 0.12.16 (protocol 14), 0.12.17 through 0.12.19 (protocol 15), 0.12.20 through 0.12.
|
|
1859
|
+
| 0.7.0 through 0.12.0 (protocol 10), 0.12.1 and 0.12.2 (protocol 11), 0.12.3 (protocol 12), 0.12.4 through 0.12.15 (protocol 13), 0.12.16 (protocol 14), 0.12.17 through 0.12.19 (protocol 15), 0.12.20 through 0.12.30 (protocol 16) | the target's, through `bunx` as above |
|
|
1639
1860
|
| any supported source, with the installed CLI already at the target | `ahub upgrade` below, which is the same coordinator |
|
|
1640
1861
|
| 0.5.x or earlier (protocol 8 and older) | not supported: bootstrap by hand with the matching CLI |
|
|
1641
1862
|
|
|
@@ -1667,14 +1888,14 @@ projects first:
|
|
|
1667
1888
|
|
|
1668
1889
|
```bash
|
|
1669
1890
|
ahub restart --dry-run
|
|
1670
|
-
ahub upgrade --to 0.12.
|
|
1891
|
+
ahub upgrade --to 0.12.30 --dry-run
|
|
1671
1892
|
```
|
|
1672
1893
|
|
|
1673
1894
|
Apply only after reviewing the plan:
|
|
1674
1895
|
|
|
1675
1896
|
```bash
|
|
1676
1897
|
ahub restart --yes
|
|
1677
|
-
ahub upgrade --to 0.12.
|
|
1898
|
+
ahub upgrade --to 0.12.30 --yes
|
|
1678
1899
|
ahub recovery status [<operation-id>]
|
|
1679
1900
|
ahub recovery resume [<operation-id>]
|
|
1680
1901
|
ahub recovery abort [<operation-id>]
|
|
@@ -1739,7 +1960,9 @@ release. The global CLI is promoted last.
|
|
|
1739
1960
|
|
|
1740
1961
|
A lost commit or start reply is reconciled against the actual daemon identity,
|
|
1741
1962
|
package digest, terminal binding, and phase receipt. An uncertain terminal
|
|
1742
|
-
creation is never repeated blindly.
|
|
1963
|
+
creation is never repeated blindly. Recovery retains the exact-pinned first-party memory package with the preserved runtime, includes its identity and bytes in the digest, and refuses missing, symlinked or changed retained dependencies instead of resolving a mutable ancestor/global installation. It does not copy the full dependency tree.
|
|
1964
|
+
|
|
1965
|
+
After creating a managed terminal, the coordinator saves its binding before waiting up to 90 seconds for authenticated attachment to the expected target hub and conversation. An idle terminal alone is not attachment proof. Timeout retains the terminal and its diagnostic receipt; resume waits on that same binding rather than opening a duplicate. The zero-turn Claude exception still applies only when no original transcript exists. If Claude shows its development-channel confirmation screen, confirm it in
|
|
1743
1966
|
the captured terminal, then run `ahub recovery resume <operation-id>`; resume must not open a duplicate terminal.
|
|
1744
1967
|
Kimi and local sessions may start fresh with preserved routing and task context.
|
|
1745
1968
|
A Claude session that never persisted a transcript (zero turns) also starts fresh:
|
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
# Agent-mem P0 failure registry
|
|
2
|
+
|
|
3
|
+
Issue #412 P0 annex. Inventory: `docs/plans/amem-cases.json` (60 cases, 8 modes). Source references were relocated against HEAD `5af9eb9bff85f4705a351441ea7d3ff1553cf121`; case status is the inventory status, not a new verification claim. This document records retained source assertions and declared ceilings; tests and live legs were not executed for the mapping.
|
|
4
|
+
|
|
5
|
+
Evidence scope is repo-only: all mechanism and regression references below come from this checkout; external historical outcomes are retained as inventory status, not newly verified.
|
|
6
|
+
|
|
7
|
+
## Table A
|
|
8
|
+
|
|
9
|
+
Owner names describe the mechanism after the split. P4 ports core regressions; P5 retains hub integration regressions. Mixed rows can cite both. A `ceiling` citation records the first words of a current `ponytail:` comment and does not claim the limitation is fixed. S04 has no planned phase because a consensus broker is out of scope. I06 preserves the exact template source, including substitutions, for its 12 parameterized tests.
|
|
10
|
+
|
|
11
|
+
| id | mode | status | owner module | phase | retained regression or declared ceiling |
|
|
12
|
+
| --- | --- | --- | --- | --- | --- |
|
|
13
|
+
| T01 | Truth/lifecycle | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review fixes: deadlines are fixed when written, ids resolve within the project's view, and a cut-short settlement finishes at the next start` |
|
|
14
|
+
| T02 | Truth/lifecycle | regression-covered | agent-hub: tasks.ts | P5 | `test/memory-native.test.ts` :: `a late note on an already approved task starts its clock at once (review of #352)`; `test/memory-store.test.ts` :: `review of #352: a late short-term note on a settled task gets its own clock, and a forgotten replacement shows the old entry again` |
|
|
15
|
+
| T03 | Truth/lifecycle | regression-covered | agent-hub: tasks.ts | P5 | `test/memory-native.test.ts` :: `an explicit short horizon on an open task is never promoted, and a bad short_ttl_days fails the config (review of #352)`; `test/memory-store.test.ts` :: `review of #352: an explicit short note on an open task keeps its own deadline through approval` |
|
|
16
|
+
| T04 | Truth/lifecycle | regression-covered | agent-hub: tasks.ts | P5 | `test/memory-native.test.ts` :: `an approval a stop cut off before its settle entry is settled when the hub starts again (review of #352)`; `test/memory-store.test.ts` :: `review fixes: deadlines are fixed when written, ids resolve within the project's view, and a cut-short settlement finishes at the next start` |
|
|
17
|
+
| T05 | Truth/lifecycle | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #352: a demoted copy takes its hidden original with it, and malformed synced lines are skipped`; `test/memory-store.test.ts` :: `review of #352: twice-promoted chains, non-string metadata and provenance nodes` |
|
|
18
|
+
| T06 | Truth/lifecycle | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #352: a superseded source of a settled task is kept for when its replacement goes, and a corrupt machine id fails the store`; `test/memory-store.test.ts` :: `review of #352: a held source moves to long-term as soon as its replacement is forgotten`; `test/memory-store.test.ts` :: `review of #352: a held source is promoted when its replacement expires while the store is idle, and an unreadable expiry is skipped` |
|
|
19
|
+
| T07 | Truth/lifecycle | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #352: a promoted replacement keeps its supersession, and a promotion is dated as its source`; `test/memory-store.test.ts` :: `review of #352: promoting a replaced entry keeps it replaced, and a non-UTC at is skipped` |
|
|
20
|
+
| T08 | Truth/lifecycle | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #352: a supersession that would close a cycle is refused, and an undated settle is skipped`; `test/memory-store.test.ts` :: `review of #352: a cycle made by supersessions written apart drops its earliest relation` |
|
|
21
|
+
| T09 | Truth/lifecycle | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #352: a later supersede op wins over an earlier note's own supersedes` |
|
|
22
|
+
| T10 | Truth/lifecycle | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #352: a note on a task never approved leaves after the fixed backstop, and the CLI screens what it prints` |
|
|
23
|
+
| T11 | Truth/lifecycle | guard/design | agent-mem: import.ts | P4 | `test/memory-phase3.test.ts` :: `claude-mem import copies the hub's notes of this project once, never model answers, PII or secrets, and never writes claude-mem`; `test/memory-phase3.test.ts` :: `model-suggested edges are drawn but are not evidence until a person confirms them; a rejected one disappears` |
|
|
24
|
+
| D01 | Duplicate/identity | fixed-source | agent-mem: import.ts | P4 | `test/memory-sync.test.ts` :: `CLI claude-mem imports use text-only nodes across checkouts and local deny policies, then converge in either sync order` |
|
|
25
|
+
| D02 | Duplicate/identity | fixed-source | agent-mem: store.ts | P4 | `test/memory-dedupe.test.ts` :: `legacy dedupe lists without writes, applies append-only, repeats without writes, and forget removes the whole tuple`; `test/memory-dedupe.test.ts` :: `dedupe sends a single remaining head to the newest valid outside terminal, preserving both personal replacements`; `test/memory-sync.test.ts` :: `legacy duplicate dedupe syncs append-only and divergent prior relations converge after a re-run` |
|
|
26
|
+
| D03 | Duplicate/identity | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #352: two machines promoting the same demoted entry write one copy` |
|
|
27
|
+
| D04 | Duplicate/identity | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #352: a short-term id kept past its task's approval reaches the promoted copy`; `test/memory-store.test.ts` :: `review of #352: another machine with only the synced copy resolves the original short-term id` |
|
|
28
|
+
| R01 | Retrieval/capture | guard/design | agent-mem: recall.ts | P4 | `test/recall.test.ts` :: `kimi gets every platform in one call; codex gets the other platforms only, without the repeated legend` |
|
|
29
|
+
| R02 | Retrieval/capture | regression-covered | agent-mem: recall.ts | P4 | `test/recall.test.ts` :: `the token budget is split per platform so the first one cannot crowd out the rest` |
|
|
30
|
+
| R03 | Retrieval/capture | guard/design | agent-mem: client.ts | P4 | `test/memory.test.ts` :: `fail-open: a stopped worker, an unknown endpoint and a hang all yield undefined with one log line` |
|
|
31
|
+
| R04 | Retrieval/capture | historical-fixed | agent-hub: capture.ts | P5 | `test/local-worker.test.ts` :: `claude-mem capture: init, one observation per tool call, summarize, session-end; skip list and denylisted paths stay out; a down worker blocks nothing` |
|
|
32
|
+
| R05 | Retrieval/capture | historical-fixed | agent-hub: capture.ts | P5 | gap |
|
|
33
|
+
| R06 | Retrieval/capture | historical-fixed | agent-hub: bus.ts | P5 | `test/bus.test.ts` :: `shared notes ride in the preface after recall, never as a delivery of their own; the newest ten are kept` |
|
|
34
|
+
| R07 | Retrieval/capture | historical-fixed | agent-hub: bus.ts | P5 | `test/bus.test.ts` :: `a preface made while a delivery is in flight survives that delivery's failure, and both go out once` |
|
|
35
|
+
| R08 | Retrieval/capture | historical-fixed | agent-hub: templates/AGENTS.block.md | P5 | gap |
|
|
36
|
+
| R09 | Retrieval/capture | historical-partial | external: claude-mem | external | gap |
|
|
37
|
+
| R10 | Retrieval/capture | historical-partial | external: claude-mem | external | gap |
|
|
38
|
+
| R11 | Retrieval/capture | historical-partial | external: claude-mem | external | gap |
|
|
39
|
+
| R12 | Retrieval/capture | historical-fixed | external: claude-mem | external | gap |
|
|
40
|
+
| R13 | Retrieval/capture | fixed-source | agent-mem: import.ts | P4 | `test/memory-phase3.test.ts` :: `claude-mem project filtering prevents starvation and the local cursor advances bounded windows, including skipped rows`; `test/memory-phase3.test.ts` :: `claude-mem cursor survives reopen and resets a recreated project source even when unrelated rows have higher ids`; `test/memory-phase3.test.ts` :: `claude-mem cursor records completed rows after a callback error and restart reconsiders earlier privacy skips` |
|
|
41
|
+
| X01 | Project isolation | open-ceiling | external: claude-mem | external | ceiling `src/memory/import.ts` :: `ponytail: claude-mem keys projects by` |
|
|
42
|
+
| X02 | Project isolation | regression-covered | agent-mem: import.ts | P4 | `test/memory-phase3.test.ts` :: `claude-mem import copies the hub's notes of this project once, never model answers, PII or secrets, and never writes claude-mem` |
|
|
43
|
+
| X03 | Project isolation | regression-covered | agent-mem: recall.ts | P4 | `test/memory-store.test.ts` :: `review fixes: deadlines are fixed when written, ids resolve within the project's view, and a cut-short settlement finishes at the next start`; `test/memory-store.test.ts` :: `review of #352: projects are keyed by their origin remote, or by name and directory without one`; `test/recall.test.ts` :: `worktrees use the native parent and parent/worktree aliases` |
|
|
44
|
+
| X04 | Project isolation | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #352: a replacement never narrows who sees an entry` |
|
|
45
|
+
| P01 | Privacy | fixed-source | agent-hub: tasks.ts | P5 | `test/tasks.test.ts` :: `a done summary, review note or unmet item matching a PII pattern reaches neither claude-mem nor a cloud peer`; `test/tasks.test.ts` :: `the completed-change notice leaves out a summary line and file names that match a PII pattern` |
|
|
46
|
+
| P02 | Privacy | fixed-source | agent-hub: adapters/local-worker.ts | P5 | `test/local-worker.test.ts` :: `hub tool capture cannot leak refused notes, withheld summaries or PII proposals`; `test/local-worker.test.ts` :: `refused hub_send text is not captured while ordinary file tools still are` |
|
|
47
|
+
| P03 | Privacy | fixed-source | agent-hub: daemon.ts | P5 | `test/local-worker.test.ts` :: `daemon local board lists expose PII rows only in a PII turn, never ordinary history or answers` |
|
|
48
|
+
| P04 | Privacy | regression-covered | agent-hub: tasks.ts | P5 | `test/memory-native.test.ts` :: `a note about a private path is refused before either memory sees it (review of #352)`; `test/memory-store.test.ts` :: `review of #352: notes leaving for a model are screened under the current PII policy, and a replaced promoted copy stays replaced elsewhere` |
|
|
49
|
+
| P05 | Privacy | regression-covered | agent-mem: recall.ts | P4 | `test/memory-store.test.ts` :: `review of #352: notes leaving for a model are screened under the current PII policy, and a replaced promoted copy stays replaced elsewhere` |
|
|
50
|
+
| P06 | Privacy | regression-covered | agent-hub: cli/memory.ts | P5 | `test/memory-store.test.ts` :: `review of #352: a note on a task never approved leaves after the fixed backstop, and the CLI screens what it prints` |
|
|
51
|
+
| P07 | Privacy | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #352: every promotion goes through the screen: a note it keeps stays short-term, another project's waits`; `test/memory-store.test.ts` :: `review of #352: the promotion screen ignores a task key's numbers, sees the task's current paths, and a refused promote says why` |
|
|
52
|
+
| P08 | Privacy | guard/design | agent-mem: sync.ts | P4 | `test/memory-sync.test.ts` :: `a public remote is refused, an unknown one warns, and a store with no remote creates nothing` |
|
|
53
|
+
| P09 | Privacy | fixed-source | agent-hub: config-trust.ts | P5 | `test/config-trust.test.ts` :: `a committed config keeps the defaults for every machine-local field, logs them, and still applies the rest` |
|
|
54
|
+
| I01 | Index integrity/freshness | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `an incremental index equals a rebuild from the logs, in any order, and bad lines are skipped`; `test/memory-store.test.ts` :: `review of #352: a demoted copy takes its hidden original with it, and malformed synced lines are skipped`; `test/memory-store.test.ts` :: `review of #352: promoting a replaced entry keeps it replaced, and a non-UTC at is skipped`; `test/memory-sync.test.ts` :: `review of #354: a lock a killed git left behind never hides this machine's entries, a poisoned goal line is skipped, and goals pass the PII screen` |
|
|
55
|
+
| I02 | Index integrity/freshness | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #352: a segment that shrinks rebuilds the index, and two hops pass through provenance`; `test/memory-store.test.ts` :: `review of #352: a segment rewritten in place without shrinking rebuilds the index`; `test/memory-store.test.ts` :: `review of #352: a rewrite past the first 4 KiB is caught, and mem: ids resolve` |
|
|
56
|
+
| I03 | Index integrity/freshness | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #352: a note written while the index is locked shows on the next read` |
|
|
57
|
+
| I04 | Index integrity/freshness | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #352: a note written while the index is locked shows on the next read` |
|
|
58
|
+
| I05 | Index integrity/freshness | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #352: a long-lived store reads what another process wrote, with no write of its own` |
|
|
59
|
+
| I06 | Index integrity/freshness | fixed-source | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #378: deferred ${change} is indexed by the next ${anotherReader ? "other store's" : "writer's"} relevant read` |
|
|
60
|
+
| I07 | Index integrity/freshness | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #352: a read prunes what expired while the store sat idle, so the index equals a rebuild` |
|
|
61
|
+
| S01 | Sync/concurrency | regression-covered | agent-mem: store.ts | P4 | `test/memory-store.test.ts` :: `review of #352: a newer entry's graph shows what it supersedes, and a fresh home gets one machine id`; `test/memory-store.test.ts` :: `review of #352: a superseded source of a settled task is kept for when its replacement goes, and a corrupt machine id fails the store` |
|
|
62
|
+
| S02 | Sync/concurrency | regression-covered | agent-mem: sync.ts | P4 | `test/memory-sync.test.ts` :: `review of #354: a lock a killed git left behind never hides this machine's entries, a poisoned goal line is skipped, and goals pass the PII screen` |
|
|
63
|
+
| S03 | Sync/concurrency | guard/design | agent-mem: sync.ts | P4 | `test/memory-sync.test.ts` :: `two machines sync through a private remote: each sees the other's long-term entries, short-term ones stay home, writes never conflict and a forget propagates`; `test/memory-sync.test.ts` :: `a public remote is refused, an unknown one warns, and a store with no remote creates nothing` |
|
|
64
|
+
| S04 | Sync/concurrency | open-ceiling | agent-mem: store.ts | none (declared ceiling) | ceiling `src/memory/store.ts` :: `ponytail: advisory, eventually consistent through` |
|
|
65
|
+
| C01 | Scale/cost/measurement | open-ceiling | agent-mem: store.ts | P6 | ceiling `src/memory/store.ts` :: `ponytail: the check and the append are not one step across processes` |
|
|
66
|
+
| C02 | Scale/cost/measurement | open-ceiling | agent-mem: store.ts | P6 | ceiling `src/memory/store.ts` :: `ponytail: hashes the whole indexed prefix on every read` |
|
|
67
|
+
| C03 | Scale/cost/measurement | open-ceiling | agent-mem: store.ts | P6 | ceiling `src/memory/store.ts` :: `ponytail: a few retries; a store with more writers racing on one segment` |
|
|
68
|
+
| C04 | Scale/cost/measurement | open-ceiling | agent-mem: store.ts | P6 | ceiling `src/memory/store.ts` :: `ponytail: computed in memory over all rows` |
|
|
69
|
+
| C05 | Scale/cost/measurement | open-ceiling | agent-mem: store.ts | P9 | ceiling `src/memory/store.ts` :: `ponytail: pruning trusts the wall clock` |
|
|
70
|
+
| C06 | Scale/cost/measurement | open-ceiling | agent-mem: recall.ts | P1 | ceiling `src/memory/recall.ts` :: `ponytail: 3 characters per token` |
|
|
71
|
+
| C07 | Scale/cost/measurement | open-ceiling | agent-hub: daemon.ts | P6 | ceiling `src/hub/daemon.ts` :: `ponytail: recomputed on every poll from the whole index` |
|
|
72
|
+
| C08 | Scale/cost/measurement | open-measurement | agent-hub: research.ts | P10 | gap |
|
|
73
|
+
|
|
74
|
+
## Gaps
|
|
75
|
+
|
|
76
|
+
A gap is an absent regression for the specific failure, not an assertion that its mechanism is absent. External gaps require upstream or provider evidence; no agent-mem phase can establish them alone.
|
|
77
|
+
|
|
78
|
+
- R05 (P5): The capture integration checks receipt after stop, but does not assert shutdown awaits the bounded end request before process exit. Add the regression in P5.
|
|
79
|
+
- R08 (P5): Rendering tests check workflow kinds, but do not assert generated instructions interpret only presence as reference. Add the regression in P5.
|
|
80
|
+
- R09 (external): No repository test reproduces observer prose instead of XML or establishes dropped-data recovery. Obtain an upstream/provider retained regression; ownership remains external.
|
|
81
|
+
- R10 (external): No repository test establishes real observer allowance/provider recovery. Obtain an upstream/provider retained regression; ownership remains external.
|
|
82
|
+
- R11 (external): No repository test establishes installed plugin/cache/home readiness for the external worker. Obtain an upstream/provider retained regression; ownership remains external.
|
|
83
|
+
- R12 (external): No repository test reproduces the launcher compilation versus SessionStart race. Obtain an upstream/provider retained regression; ownership remains external.
|
|
84
|
+
- C08 (P10): Counts/report plumbing does not encode causal adoption effectiveness, manual retry labels, or the field window. Add the regression in P10.
|
|
85
|
+
|
|
86
|
+
## Table B
|
|
87
|
+
|
|
88
|
+
All live legs are **unrun**. Fixture assertions and this validator cannot close the claims below. Line citations point to the unchecked live requirements in `docs/smoke.md` at the HEAD above.
|
|
89
|
+
|
|
90
|
+
| id | live leg | owner | phase | status | claim blocked |
|
|
91
|
+
| --- | --- | --- | --- | --- | --- |
|
|
92
|
+
| L01 | Two physical Macs, private GitHub SSH remote; entries, forget, lease and conflict read back on both | person, claude records | before #258 closes | unrun | Physical cross-machine sync and conflict behavior, `docs/smoke.md:63` |
|
|
93
|
+
| L02 | gh reports an actual public test remote and sync refuses it | person | before #258 closes | unrun | Actual public-remote refusal, `docs/smoke.md:64` |
|
|
94
|
+
| L03 | Import the real local claude-mem with counts/skips; source DB unchanged | claude | P5 | unrun | Real import and source immutability, `docs/smoke.md:65` |
|
|
95
|
+
| L04 | A reachable real model emits dashed suggestions; briefs exclude them until confirmation | claude, person looks at the dashboard | P5 | unrun | Live suggestions and evidence exclusion, `docs/smoke.md:66` |
|
|
96
|
+
| L05 | Memory and Long-term tasks tabs, light/dark, phone width, browser | person | P5 | unrun | Human browser inspection, `docs/smoke.md:67` |
|
|
97
|
+
| L06 | off / claude-mem / native memory arms, five repeats, plus two weeks of field records (#258 AC11) | claude runs the arms by hand | P10 | unrun | Controlled memory-arm and field effectiveness, `docs/smoke.md:68` |
|
|
98
|
+
| L07 | Native Claude/Codex attach with tooling event and installed add-ons (#248) | claude | P10 | unrun | Actual installed runtime fingerprint, `docs/smoke.md:97` |
|
|
99
|
+
| L08 | Interleaved randomized add-on arms for claude-mem, ponytail, ripwire, five repeats (#248 AC2/3) | claude | P10 | unrun | Controlled add-on comparisons, `docs/smoke.md:98` |
|
|
100
|
+
| L09 | Two weeks of research.enabled, then the add-ons report (#248) | person enables, claude reports | P10 | unrun | Field observation window and report, `docs/smoke.md:99` |
|
|
101
|
+
| L10 | Revise the add-on decisions from measured results | person | after P10 | unrun | Evidence-backed adoption decisions, `docs/smoke.md:100` |
|
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
# Agent-mem P0 baseline
|
|
2
|
+
|
|
3
|
+
Issue #412, Evidence E1-E12 and P0. This snapshot records what the sources reproduce, including disagreements with the plan. The [failure registry](2026-10-16-amem-failure-registry.md) is the companion P0 annex; its retained source assertions do not certify live legs.
|
|
4
|
+
|
|
5
|
+
# Capture contract
|
|
6
|
+
|
|
7
|
+
- Capture date: 2026-10-16. Source HEAD: `8f59603e3637419465b548b6e6a073b5b525b164` (before the receipt-second correction). The source references below were checked at this HEAD.
|
|
8
|
+
- SQLite observation and work-state cutoff: `created_at_epoch < 2026-10-16T00:00:00.000Z`, in epoch milliseconds. Each collector invocation uses one read transaction on a `{ readonly: true }` connection.
|
|
9
|
+
- Native log cutoff: `at <= 2026-10-15T21:53:00.000Z`, over `log/*.jsonl` and `local/*.jsonl`. Counts are raw dated log rows, not current visibility or an index projection. A promoted copy counts as another row. `by_tier` uses the row's declared tier; ops without a tier are `other`. `segments` counts physical placement.
|
|
10
|
+
- Event and recall cutoff: `at < 2026-10-15T22:35:36.000Z` (the entire receipt second). Only `events*.jsonl` and the current `hub.log` are read; missing rotated log archives are not inferred.
|
|
11
|
+
- Auto-memory/Codex file counts and research-directory presence have no trustworthy historical cutoff and are current directory counts. Symlink directories are not followed. Claude `project_dirs` counts every immediate directory, while `projects_with_memory` counts directories with at least one eligible note. Codex counts regular files recursively, including non-Markdown files.
|
|
12
|
+
- Fail distinctness is SHA-256 of exact UTF-8 `text`, not title or entry id. Hashes and note text are never printed. Different tier copies with identical text count once.
|
|
13
|
+
- No native index, machine identity file, or vault file is opened. No model, sync, write API or subprocess is called by the collector. Errors are numeric and do not expose source paths or messages. `readable: 0` is unavailable, never zero data; `null` marks evidence outside this script's scope.
|
|
14
|
+
- All paths have CLI overrides: `--claude-mem-db`, `--memory-dir`, `--project-dir`, `--claude-projects`, `--codex-memories`, `--research-dir`, `--cases`. Defaults use the home directory and the script's checkout. For this capture, `--project-dir` targeted the main agent-hub checkout, since this worktree has no runtime state. Pass `--project-dir` pointing to the runtime checkout when collecting from a worktree. In the runtime checkout, the default command reads its own state.
|
|
15
|
+
- Bare `bun scripts/amem/baseline.ts` in this worktree exits 0; E4 and E5.briefs report unreadable because this worktree has no `.agenthub/state`. The explicit-project capture below reads those sources successfully. No source was unreadable from codex sandbox with that override.
|
|
16
|
+
|
|
17
|
+
# Evidence table
|
|
18
|
+
|
|
19
|
+
| Id | Number and unit | Cutoff | Script key or source query | Status |
|
|
20
|
+
| --- | --- | --- | --- | --- |
|
|
21
|
+
| E1 | 200,717 observations; 4,882,886,254 discovery tokens; 14 days: min 849, max 11,749, median 5,411, total 82,067 | SQLite cutoff above; daily interval [2026-10-02, 2026-10-16) UTC | `E1`; `count(*)`, `sum(discovery_tokens)`, and 14 daily half-open epoch ranges over observations | drifting: historical rows can later be inserted, merged or deleted; daily counts reproduced |
|
|
22
|
+
| E1 | Prior receipt: 11 GB store, 7.1 GB vectors | Earlier measurement, no size cutoff retained | #412 E1; no recursive disk scan performed here | drifting; not remeasured |
|
|
23
|
+
| E2 | 0 rows with relevance_count > 0; 0 with occurrence_count > 1 | SQLite cutoff | `E2`; conditional sums over observations | drifting; zero is a missing use signal, not evidence of no use |
|
|
24
|
+
| E3 | 128 rows: 41 log, 87 local; 102 notes, 26 settle ops; 52 decision, 44 finding, 3 contract, 3 fail; decisions 26 approved / 24 changes requested / 2 other; long decisions 38, auto outcomes 36 | Native cutoff | `E3`; anchored first-line text regexes and declared tier | rebuildable from dated logs; all frozen native receipts reproduced |
|
|
25
|
+
| E4 | One witnessed channel delivery duplicated the presence block; 15 displayed lines reportedly all auto outcomes cut mid-sentence | Original witnessed delivery, no retained machine-readable delivery receipt | #412 E4, no source in script for the duplicated block or its cause | witnessed-only; not independently reproduced |
|
|
26
|
+
| E4 | 17 recall log lines, 49,135 chars total, mean 2,890.294117647059 chars | Event/recall cutoff | `E4`; timestamped `recall <peer>: N chars` lines | rebuildable from retained current log; 7-start receipt not counted by this script |
|
|
27
|
+
| E5 | 3 fail note rows, 2 distinct text hashes; 11 briefs, 61 native items, 24 external capture items, 5 fails, 5 briefs with fails > 0 | Native cutoff for notes; event cutoff for briefs | `E5.notes`, `E5.briefs`; memory_brief event sums | rebuildable; frozen brief receipt reproduced |
|
|
28
|
+
| E6 | 144 project keys; 132 without slash; exact agent-hub key 10,673 rows, scoped agent-hub key 28,369 rows | SQLite cutoff | `E6`; count distinct project, instr(project,'/'), exact-key conditional sums | drifting; key counts reproduce, exact-key row counts differ |
|
|
29
|
+
| E7 | Prior Pi denial receipts #69 and #73 | Earlier sandbox runs | #412 E7; `E7.sandbox_receipt` is null | witnessed-only; no new Pi sandbox or graph/rebuild refusal run |
|
|
30
|
+
| E8 | 165 Claude note files (excluding MEMORY.md), 217 immediate project directories, 8 with notes; 257 recursive Codex files; 3,916 work-state entries; 102 native notes | File counts at capture; SQLite cutoff for work-state; native cutoff for notes | `E8`, `E3.by_op.note` | drifting for files/SQLite; rebuildable native notes |
|
|
31
|
+
| E8 | 813 vault notes, previously obtained through Obsidian MCP | Earlier #412 snapshot | #412 E8; `E8.vault` is null; vault remains MCP-only | witnessed-only for this capture; not rebuildable by this script |
|
|
32
|
+
| E9 | Fresh-store single runs at N=2,000 / 20,000 / 200,000; full stdout below | New run per N, 2026-10-16 | `bun scripts/amem/scale-bench.ts N` | rebuildable workload; timings vary, no latency-distribution claim |
|
|
33
|
+
| E10 | Unknown ops ignored; unknown note kinds/scopes skipped; unknown optional fields hashed but omitted on projection/promotion | Source HEAD above | `src/memory/store.ts:466` note validation, `:468` explicit projection, `:479` recognized-op branch to `:484`, `:338` promotion field list; `entryId` at `:73` | static; line references rechecked |
|
|
34
|
+
| E11 | 0 research directories at configured default location | Capture time | `E11.research_directory_exists`; no study records inferred from directory absence | drifting; #412 reports the AC11/add-on arms as unrun, not independently audited here |
|
|
35
|
+
| E12 | 60 cases, 8 modes; 28 regression-covered, 9 open-ceiling, 8 fixed-source, 6 historical-fixed, 5 guard/design, 3 historical-partial, 1 open-measurement | Frozen cases file hash below | `E12`; fixed public mode/status taxonomy; full mode counts below | rebuildable inventory, not proof that all cases are fixed |
|
|
36
|
+
|
|
37
|
+
# Differences from #412 and frozen receipts
|
|
38
|
+
|
|
39
|
+
No value was tuned to match a receipt. During the reviewer-requested stdout re-freeze, the current recursive Codex file count moved from 124 to 257; this uncut file count remains drifting. These are all mismatches or non-comparable aggregates observed in this capture; unmatched historical claims remain qualified above.
|
|
40
|
+
|
|
41
|
+
| Measurement | #412 or receipt | This capture | Interpretation |
|
|
42
|
+
| --- | --- | --- | --- |
|
|
43
|
+
| Observations | 200,467 | 200,717 | SQLite cutoff is frozen, source rows remain mutable; cause of 250-row difference not established |
|
|
44
|
+
| Discovery tokens | 4.87 B (rounded) | 4,882,886,254 (4.882886254 B) | Does not round to 4.87 B; no exact old sum retained here |
|
|
45
|
+
| Recall chars mean | about 2,990 | 2,890.294117647059 | 17 line count matches; prior average not reproduced |
|
|
46
|
+
| Exact agent-hub project observations | 10,661 | 10,673 | Same exact-key query with frozen cutoff; mutable source difference |
|
|
47
|
+
| Scoped agent-hub project observations | 28,148 | 28,369 | Same exact-key query with frozen cutoff; mutable source difference |
|
|
48
|
+
| Claude auto-memory files | 172 | 165 | Current immediate memory/*.md files excluding MEMORY.md |
|
|
49
|
+
| Claude project count | 37 projects | 217 immediate directories; 8 with eligible notes | Old inclusion rule not retained, so neither count is claimed comparable |
|
|
50
|
+
| Codex memory files | 7 | 257 recursive regular files | Old inclusion rule not retained; current definition includes nested and non-Markdown files |
|
|
51
|
+
| Work-state entries | 3,888 | 3,916 | Frozen created_at cutoff, mutable source |
|
|
52
|
+
| 20K open / search / 2-hop ms | 347 / 74 / 50 | 875 / 244 / 149 | Different environment and single samples; no regression conclusion |
|
|
53
|
+
| 200K open / search / 2-hop ms | 3,906 / 835 / 506 | 12,877 / 2,609 / 1,495 | Different environment and single samples; no regression conclusion |
|
|
54
|
+
| 200K index MB | 405 (rounded) | 404.9 | Rounds to the same whole-MB value |
|
|
55
|
+
|
|
56
|
+
The daily 849 / 11,749 / 5,411 / 82,067 receipt, native 128 and 52 = 26 + 24 + 2 receipt, long 38 / 36 receipt, fail 3 / 2 receipt, brief 11 / 61 / 24 / 5 / 5 receipt, E2 zeros, E6 144 / 132, E8 102 native notes, and E12 60 / 8 reproduce.
|
|
57
|
+
|
|
58
|
+
# Frozen collector stdout
|
|
59
|
+
|
|
60
|
+
The following is the exact single stdout line, with its final LF included in the hash. Numeric errors and null placeholders are part of its schema; no note content, titles, identifiers or private project labels are exported.
|
|
61
|
+
|
|
62
|
+
```json
|
|
63
|
+
{"E1":{"observations":200717,"discovery_tokens":4882886254,"daily_utc":{"2026-10-02":849,"2026-10-03":6488,"2026-10-04":3406,"2026-10-05":8510,"2026-10-06":2385,"2026-10-07":4564,"2026-10-08":4837,"2026-10-09":9446,"2026-10-10":10661,"2026-10-11":11749,"2026-10-12":6707,"2026-10-13":5985,"2026-10-14":4220,"2026-10-15":2260},"days":14,"min":849,"max":11749,"median":5411,"total":82067},"E2":{"relevance_positive":0,"occurrence_repeated":0},"E3":{"rows":128,"malformed":0,"segments":{"log":41,"local":87},"by_op":{"note":102,"goal":0,"forget":0,"supersede":0,"demote":0,"settle":26,"goal_state":0,"goal_link":0,"lease":0,"lease_force":0,"suggest":0,"confirm":0,"reject":0,"unknown":0},"by_kind":{"decision":52,"finding":44,"contract":3,"fail":3},"by_tier":{"short":61,"long":41,"other":26},"decisions":{"approved":26,"changes_requested":24,"other":2},"long_decisions":38,"long_auto_outcomes":36},"E4":{"readable":1,"recalls":17,"chars_total":49135,"mean_chars":2890.294117647059},"E5":{"notes":{"fail_rows":3,"distinct_fail_text_hashes":2},"briefs":{"readable":1,"briefs":11,"native_items":61,"claude_mem_items":24,"fails":5,"with_fails":5,"malformed":0}},"E6":{"project_keys":144,"keys_without_slash":132,"agent_hub":10673,"work_dev_agent_hub":28369},"E7":{"sandbox_receipt":null},"E8":{"claude":{"readable":1,"files":165,"project_dirs":217,"projects_with_memory":8},"codex":{"readable":1,"files":257},"claude_mem":{"work_state_entries":3916},"vault":null},"E9":{"benchmark":null},"E10":{"static_code_review":null},"E11":{"readable":1,"research_directory_exists":0},"E12":{"readable":1,"cases":60,"modes":8,"by_mode":{"Truth/lifecycle":11,"Duplicate/identity":4,"Retrieval/capture":13,"Project isolation":4,"Privacy":9,"Index integrity/freshness":7,"Sync/concurrency":4,"Scale/cost/measurement":8,"other":0},"by_status":{"regression-covered":28,"open-ceiling":9,"fixed-source":8,"historical-fixed":6,"guard/design":5,"historical-partial":3,"open-measurement":1,"other":0}}}
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
# Scale runs
|
|
67
|
+
|
|
68
|
+
Each N ran once in a fresh Bun process and a new temporary home/index, sequentially in the remote development container without a parallel benchmark worker pool. The original workload is unchanged, including its open → search → graph → rebuild order. "Cold" means no pre-existing derived database for the open measurement; OS page caches were not flushed. Search/graph run after opening and are not separately cold. There are no repeated samples or p95 values. `logMB` and `indexMB` use decimal MB. Only synthetic temporary data was written.
|
|
69
|
+
|
|
70
|
+
Environment: Linux remote development container, Bun 1.4.2. The configured container allowance is 16 CPUs / 32 GB; the script is single-process with no worker pool. No benchmark worker concurrency was added. These runs cannot be compared as a speed regression against the original host's one-shot receipts.
|
|
71
|
+
|
|
72
|
+
```json
|
|
73
|
+
{"N":2000,"logMB":2.6,"indexMB":4.1,"openMs":91,"searchMs":23,"hits":50,"graph2Ms":15,"graphNodes":22,"rebuildMs":67}
|
|
74
|
+
{"N":20000,"logMB":26.3,"indexMB":40.4,"openMs":875,"searchMs":244,"hits":50,"graph2Ms":149,"graphNodes":97,"rebuildMs":981}
|
|
75
|
+
{"N":200000,"logMB":263.6,"indexMB":404.9,"openMs":12877,"searchMs":2609,"hits":50,"graph2Ms":1495,"graphNodes":200,"rebuildMs":16027}
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
# Digests
|
|
79
|
+
|
|
80
|
+
| Artifact | SHA-256 |
|
|
81
|
+
| --- | --- |
|
|
82
|
+
| Collector stdout (including LF) | `2af1390b5d900726c0909b6ab817f1d605bacb855a84dcf51acf518a2a3319e6` |
|
|
83
|
+
| `scripts/amem/baseline.ts` | `cdd0d263386dd9cc8960a4a95823122fbe3c1a77eec0121ce45960065d15f626` |
|
|
84
|
+
| `docs/plans/amem-cases.json` | `1f20f6885f0e9437e88bda427499f43eb97f8e0691f615cfba28b97b11aa92a4` |
|
|
85
|
+
| `scripts/amem/scale-bench.ts` | `6d8a151e649097898394c59d6939abe1408c15785d8eaa875ea74e82588d8f55` |
|
|
86
|
+
| Three benchmark stdout lines (including LFs) | `d3f0432d1e78abf8701b21d7f3fbf95732b257aca36e83b3daa6a33f0299e045` |
|
|
87
|
+
|
|
88
|
+
To verify the stdout digest, extract the JSON block exactly with its final LF and hash its bytes. Rerunning the collector can drift even with the same cutoff, so a later differing stdout hash is not automatically script drift.
|
|
89
|
+
|
|
90
|
+
# Verification
|
|
91
|
+
|
|
92
|
+
- Count-only fixture: 24 assertions plus missing-store and invalid-argument checks passed. It covered inclusive native and exclusive end-of-second event boundaries, strict SQLite cutoff, duplicate fail text across tiers, MEMORY.md exclusion, recursive file counting, no sensitive fixture strings in output, and byte/file-set equality of fixture inputs before and after collection. The ad hoc fixture lived outside the repo and is not a retained regression.
|
|
93
|
+
- Explicit-project collector and bare-default collector both exited 0. The bare-default worktree-state limitation is documented above.
|
|
94
|
+
- `bun x tsc --noEmit`: passed in the remote container after frozen-lockfile install.
|
|
95
|
+
- Registry validator passed for all 60 rows; a separate ad hoc negative fixture confirmed rejection of duplicate/missing case ids, an unlisted gap, a missing ponytail marker, a missing literal test declaration, and invalid owner/phase fields.
|
|
96
|
+
- Initial local `scripts/check.sh` invocation (before the receipt-second correction): 2,791 pass / 4 fail, exit 1. The invocation PATH omitted `/usr/sbin`, making `lsof` unavailable to benchmark cwd/restoration checks. No source or assertions were changed to resolve those failures. Corrected-PATH focused verification passed: 63 pass / 0 fail across the three affected benchmark files. Subsequent full-gate attempts were deferred by admission before execution and are not test runs.
|
|
97
|
+
- Final P0 gate, reviewer decision 2026-10-16: the PR CI runs `scripts/check.sh` at the exact PR head on both `ubuntu-latest` and `macos-latest`, as verified in `.github/workflows/check.yml`. Both platform gates must finish with `check: OK`, and required CI (including seeded guards) must be green before reviewer approval. CI has not run at this handoff; no green CI or successful local full gate is claimed.
|
|
98
|
+
- No further local full-gate retries: the person's live remote desktop session holds admission while connected. The reviewer owns pushing the branch, opening the PR and approving only after green CI. This handoff submits the implementation for review, not a completed P0 acceptance claim.
|
|
99
|
+
|
|
100
|
+
Focused check receipts:
|
|
101
|
+
|
|
102
|
+
```text
|
|
103
|
+
baseline fixture: OK (24 assertions; missing-store and invalid-argument checks; source bytes and file set unchanged)
|
|
104
|
+
check-registry: OK (60 cases, 8 modes; gaps: R05, R08, R09, R10, R11, R12, C08)
|
|
105
|
+
63 pass
|
|
106
|
+
0 fail
|
|
107
|
+
281 expect() calls
|
|
108
|
+
Ran 63 tests across 3 files. [13.59s]
|
|
109
|
+
```
|
|
110
|
+
- Live accounts, two physical Macs, browser inspection, model suggestions and research arms were not run. The annex keeps all ten live legs unrun.
|