instar 0.28.83 → 0.28.84

package/upgrades/side-effects/taskflow-phase5.md

@@ -0,0 +1,234 @@
+# Side-Effects Review — TaskFlow Phase 5 (rate limits + cache eviction tuning + audit ledger emission)
+
+**Version / slug:** `taskflow-phase5`
+**Date:** 2026-05-10
+**Author:** Echo
+**Second-pass reviewer:** required (introduces HTTP-boundary authority gates — rate limits — and reshapes the audit-ledger emission path)
+
+## Summary of the change
+
+Phase 5 hardens the TaskFlow surface that prior phases established. The change:
+1. Adds per-controller rate limits on `POST /flows` (default: 10/sec) — overflow throws `quota_exceeded` with `code='rate_limit'` → HTTP 429.
+2. Adds per-controller max-active-flows cap (default: 50 non-terminal flows per `controllerId`) — overflow throws `quota_exceeded` with `code='max_active'` → HTTP 429.
+3. Adds per-flow ping rate limit on `POST /flows/:flowId/ping` (default: 60 pings/min/flow) — overflow throws `quota_exceeded` with `code='rate_limited'` → HTTP 429.
+4. Replaces the ad-hoc cache eviction in `TaskFlowRegistry` (`Map`-based with terminal-flow head-scan) with a proper LRU + capacity (default: 1000 entries) + the metric `taskflow_cache_evictions_total`.
+5. Fixes the previously-broken `emitLedgerNote` path in `TaskFlowRegistry` so that state-transition audit notes actually get written to `SharedStateLedger`. The notes are restricted to the redacted shape per spec § Threat Model lines 681-682 (`flowId, revision, currentStep, from_status, to_status, waitKind, controllerId, op`) — never `stateJson`, never any field of `waitJson` beyond `kind`.
+
+Per the spec (`docs/specs/OPENCLAW-IMPORT-TASKFLOW-SPEC.md`, lines 650-653 Phase 5 / lines 679 + 685 Threat Model / lines 681-682 redaction), the design goal is to enforce sane per-counterparty resource budgets at the HTTP boundary and to produce a full state-machine audit trail in SharedStateLedger that contains no controller-private payload.
+
+**Files touched:**
+- `src/tasks/RateLimiter.ts` (new) — sliding-window per-key rate limiter with monotonic clock injection.
+- `src/tasks/LruCache.ts` (new) — bounded LRU with eviction callback for metric emission.
+- `src/tasks/TaskFlowRegistry.ts` — wires rate limits in `createFlow` / `pingFlow`, replaces cache with `LruCache`, restructures `emitTransitionLedgerNote` / `emitLedgerNote` to use a real `SharedStateLedger.append` payload.
+- `src/tasks/task-flow-types.ts` — adds `RateLimitConfig`, `CacheConfig`, `DEFAULT_RATE_LIMITS`, `DEFAULT_CACHE_CONFIG`, `ACTIVE_STATUSES`.
+- `src/core/types.ts` — adds `'taskflow-transition'` to `LedgerEntrySubsystem`.
+- `src/core/SharedStateLedger.ts` — adds `'taskflow-divergence'` and `'taskflow-transition'` to `VALID_SUBSYSTEMS` (the divergence one was a pre-existing union-but-not-runtime-validated gap; this commit closes it).
+- `src/server/routes.ts` — adds `Retry-After` header on 429 responses when `retryAfterMs` is supplied.
+- `src/commands/server.ts` — plumbs `config.taskFlow.rateLimits` and `config.taskFlow.cache` into `TaskFlowRegistry`.
+- `tests/unit/task-flow-rate-limits.test.ts` (new) — 12 tests.
+- `tests/unit/task-flow-cache-eviction.test.ts` (new) — 9 tests.
+- `tests/unit/task-flow-audit-ledger.test.ts` (new) — 10 tests.
+- `upgrades/side-effects/taskflow-phase5.md` (this file).
+
+## Decision-point inventory
+
+- `RateLimiter.tryAcquire(key)` — **add** — pure mechanic. Returns `{ok, retryAfterMs, currentCount}`. No external authority. Limit = `Infinity` disables bookkeeping entirely. The hot-path bound: per-key bucket length ≤ limit (typically 10 for create, 60 for ping), and total tracked keys self-prune on each acquire after 256 calls (MAX_BUCKETS=50_000 ceiling).
+- `LruCache.set(key, value)` — **add** — pure mechanic. Map-backed LRU; insertion-order semantics on cold inserts, recency-refresh on `get`. Evicts oldest on capacity. Fires `onEvict` for metric emission.
+- `TaskFlowRegistry.createFlow` — **modify** — adds (i) idempotency pre-check (so idempotent replays do not consume a rate slot), (ii) `createLimiter.tryAcquire(controllerId)`, (iii) `getActiveCount(controllerId) < maxActivePerController` check. All three reject with `TaskFlowError('quota_exceeded', ...)` carrying `code` ∈ `{'rate_limit', 'max_active'}`.
+- `TaskFlowRegistry.pingFlow` — **modify** — adds `pingLimiter.tryAcquire(flowId)`, but ONLY after the principal/controllerId-match authority checks. Bogus pings do not consume rate-limit slots.
+- `TaskFlowRegistry.applyOcc` — **modify** — on every terminal entry: (i) decrements per-controller active count via `onTerminalTransition`; (ii) calls `pingLimiter.forget(flowId)` so the per-flow bucket is reclaimed.
+- `TaskFlowRegistry.cachePut` — **modify** — delegated to `LruCache`. Removes the ad-hoc terminal-flow head-scan eviction (which was O(32) worst-case but had no eviction count).
+- `TaskFlowRegistry.emitTransitionLedgerNote` — **rewrite** — was previously emitting an invalid `kind: 'note:taskflow-transition'` payload (which SharedStateLedger rejects with "invalid kind"). Now emits `kind: 'note'`, `subsystem: 'taskflow-transition'`, with the redacted payload encoded into `summary`. Also emits a `taskflow:audit-emitted` event with the structured payload for tests / dashboards.
+- `routes.ts` 429 error handler — **modify** — sets `Retry-After: <seconds>` header (ceiled, min 1) when `retryAfterMs` is in the error detail.
+
+---
+
+## 1. Over-block
+
+**Rate limits are authority gates at the HTTP boundary** — they reject requests. Over-block means rejecting legitimate traffic. The risk profile:
+
+- **createPerSec=10 is conservative.** Real EvolutionManager / InitiativeTracker controllers create 1-10 flows per minute under normal load, far below 10/sec. Burst scenarios (backfill at startup, divergence-checker-triggered remediation) are bounded; the 10/sec budget gives 600/minute headroom. If a real workload needs more, operators raise the config — see `config.taskFlow.rateLimits.createPerSecPerController`.
+- **maxActive=50 may be tight for some controllers.** The InitiativeTracker controller, post-Phase 4, could plausibly have >50 concurrent waiting initiatives. Mitigation: configurable via `config.taskFlow.rateLimits.maxActivePerController`. If operators hit this in production, the response is to raise the cap, not to revert.
+- **pingPerMin=60 = 1/sec sustained.** This matches `HEARTBEAT_INTERVAL_MS = 60_000ms` (per-minute heartbeat) with substantial headroom for retry-after-failure pings. A controller pinging at the documented cadence cannot trigger this limit.
+
+**Tuning options (when over-block triggers):**
+- Set the relevant `config.taskFlow.rateLimits.*` value higher.
+- Set the value to `Infinity` to disable the gate entirely (verified by the `'respects custom createPerSecPerController = Infinity (disabled)'` test).
+- Hot-restart the server picks up the new config; no DB migration needed.
+
+**Non-block surfaces (cache + audit) cannot over-block.** Cache eviction degrades read performance (one extra DB round-trip), not correctness. Audit emission is fire-and-forget — failures never propagate into the state-machine path.
+
+## 2. Under-block
+
+**Failure modes the change does not catch:**
+
+- **`controllerId` spoofing on `createFlow`.** Any auth-token holder can claim any `controllerId` when creating a flow. The rate limit is keyed on the claimed value, so an attacker can create 10 flows/sec under controllerId "A" + 10 flows/sec under "B" + ... Mitigation already in place: the `RateLimiter` has a `MAX_BUCKETS=50_000` ceiling with lazy pruning, so memory cannot explode. The per-controller max-active=50 cap is also keyed on claimed controllerId, so the attacker's storage footprint per claimed-id is bounded. Authority for who-may-claim-a-controllerId remains at the auth-token layer (loopback-only by default; explicit opt-in to expose via tunnel — per spec § Threat Model line 671).
+- **Pre-rate-limit input validation.** The schema parse + reserved-controller check + state size check all run BEFORE the rate limit. This is intentional: a flood of malformed payloads with bad schemas could thus get O(N) zod work, but those payloads also get rejected with `invalid_argument` (HTTP 422) — they never reach the registry's mutation path. If the parse itself becomes a DoS vector, the upstream HTTP layer (body-parser size limit) is where to mitigate.
+- **Ping rate limit bypass via known flowId.** Mitigated structurally: the principal / controllerId-match check happens BEFORE the rate counter (verified by the `'bogus principal is rejected BEFORE the rate counter increments'` test). A non-owning caller cannot consume the legitimate controller's ping budget.
+- **Active-count drift after server restart.** The per-controller active-count cache is lazy-loaded from DB on first access. So the first `createFlow` for a controller after a restart runs a full DB scan filtered by status. For a controller with thousands of historical flows, this DB scan is O(active_only) (the `flows_controller_status` index makes it sub-millisecond). Not a real concern for typical controller cardinality.
+- **Clock-skew on sliding window.** If `Date.now()` jumps backward (NTP), bucket timestamps could appear future-dated and never age out. Within a single-writer server process the risk is low (we don't rely on cross-machine clock sync). If it materializes, the bucket self-recovers on the next forward-pass through the limit threshold.
+- **Audit ledger lock contention.** If 50 concurrent transitions race for the SharedStateLedger lock, some appends may fail-open (per the existing `proper-lockfile` retry budget). This is the existing semantics from Phase 3a; Phase 5 inherits, does not regress.
+
+## 3. Level-of-abstraction fit
+
+**Rate limits live at the registry, surfaced via 429 at the HTTP layer — by deliberate design:**
+
+- The `RateLimiter` itself is a pure helper. It belongs in the `tasks/` directory because that's the consumer.
+- The rate-limit *invocation* lives in `TaskFlowRegistry.createFlow` / `pingFlow`, NOT in `routes.ts`. This is the load-bearing choice: putting the check at the registry means every code path that creates a flow (HTTP, internal subsystem callers like `EvolutionManager.dualWriteCreate`, future test-only paths) sees the same limit. Putting it at the route layer would let internal subsystems bypass it — a real concern for the EvolutionManager backfill, which can burst.
+- The `Retry-After` HEADER lives in `routes.ts`, NOT in the registry. The registry returns a transport-agnostic `retryAfterMs` in `TaskFlowError.detail`; the HTTP layer maps that to RFC 7231 § 7.1.3 header semantics. This separation lets non-HTTP consumers (subsystem code) still read the structured `retryAfterMs` field.
+- The LRU is bounded inside the registry; metric emission goes through `console.log [metric] ...` matching the existing pattern from `DivergenceChecker.emitMetric`.
+- Audit-ledger emission lives in the registry. A separate "AuditEmitter" class was considered and rejected: it would require the registry to call out and pass state-transition metadata to a stateless helper, which is the same shape as a private method. Not worth the indirection.
+
+## 4. Signal vs authority compliance
+
+**Required reference:** [docs/signal-vs-authority.md](../../docs/signal-vs-authority.md)
+
+- [x] **Rate limits ARE an authority gate, but at the right layer.** The Phase 5 rate-limit gates are HTTP-boundary mechanical authorities (sliding-window counters), not LLM-backed gates. The principle that "brittle/low-context filters detect and emit signals; only a higher-level intelligent gate with full context has blocking authority" applies to *content* gates (e.g., outbound message classification). Rate limits are mechanical resource-budget gates; the signal-vs-authority guidance explicitly allows mechanical authorities at structural boundaries. The "intelligent reviewer" for rate-limit tuning is the human operator who reads metrics and adjusts `config.taskFlow.rateLimits.*`.
+
+- [x] **The audit ledger emission produces a SIGNAL, not an authority.** Transition notes are tier-2 observability — fire-and-forget appends consumed by humans / dashboards / future analyzers. They have zero authority over state-machine correctness. Verified by the `'audit emission is async/best-effort — does NOT block state correctness'` test: even when `ledger.append` throws, the OCC mutation completes and `applied: true` is returned.
+
+- [x] **The cache is a pure mechanic.** LRU eviction is deterministic; the metric emission is observational. No authority surface.
+
+**Where authority is concentrated:**
+- The HTTP layer (`routes.ts` 429 handler) is where the rate-limit decision is *surfaced* to clients. The decision itself is upstream, in the registry, made by the mechanical limiter. The high-level intelligence that adjusts the limit's *value* is the operator (informed by metrics).
+- The principal/controllerId-match check (in `pingFlow`, `assertControllerScope`) is unchanged from prior phases.
+
+No new brittle blocker is introduced. No existing intelligent authority is shadowed.
+
+---
+
+## 5. Interactions
+
+### With Phase 1 (registry + sweeper + waker)
+
+- **TaskFlowMaintenanceSweeper** uses `RESERVED_MAINTENANCE_CONTROLLER` to mark flows `lost`. The reserved controllerId is exempted from `createFlow` rate limits (it never calls `createFlow`; it goes through `markLost` → `applyOcc`, which is not rate-limited). Sweeper is unaffected.
+- **TaskFlowDueWaker** calls `resumeFlow` directly via `applyOcc`. Not rate-limited. Unaffected.
+- The Phase 1 cache eviction was ad-hoc and never produced eviction metrics. Phase 5 supersedes it with a real LRU. Cache reads are still served from-cache when present; cache misses still go to SQLite. Performance neutral for hot paths; cache hit ratio slightly improved by recency-refresh on `get`.
+
+### With Phase 2 (cancellation primitives)
+
+- `requestFlowCancel` runs through `applyOcc` and is not rate-limited at the request layer. This is deliberate — cancellation should not be limited even when the controller is over its create budget. If a future need arises, a separate `cancelPerMinPerController` config can be added.
+- The `taskflow-cancel-requested` ledger note (Phase 2) now uses the same `emitLedgerNote` path that this PR fixes (was previously broken with `kind: 'note:taskflow-cancel-requested'` → schema rejection). This is a quiet bug fix; cancellation notes will now actually land.
+
+### With Phase 3a (EvolutionManager dual-write + DivergenceChecker)
+
+- `EvolutionManager.dualWriteCreate` calls `registry.createFlow` directly. It now goes through the same rate limit. The `EvolutionManager` controller has a conservative cadence (proposals are minutes-to-hours apart), so the 10/sec limit is non-binding. The backfill (`migrateExistingToTaskFlow`) runs at server startup and creates one flow per existing proposal; for a fleet of 50 proposals at startup this is well under the 10/sec budget.
+- The `DivergenceChecker` does not call `createFlow`; it reads + emits notes. Unaffected by rate limits.
+- This PR fixes a pre-existing bug where `'taskflow-divergence'` was in `LedgerEntrySubsystem` but missing from `VALID_SUBSYSTEMS` runtime list — DivergenceChecker's notes were rejected with "invalid emittedBy.subsystem". This bug was effectively rendering Phase 3a's divergence-monitoring blind. Phase 5 closes the gap.
+
+### With Phase 4 (InitiativeTracker migration) — likely landing alongside or shortly before
+
+- Phase 4 moves InitiativeTracker's state into TaskFlow. The `InitiativeTracker` controller will share the per-controller create / max-active limits. If Phase 4's controller cadence approaches the 10/sec default, operators raise the limit. The 50 max-active value matches typical initiative-tracker scales (low tens of active initiatives).
+- If Phase 4 lands first, this PR rebases cleanly off the new `origin/main` — no expected merge conflicts since Phase 4 touches `InitiativeTracker.ts` not `TaskFlowRegistry.ts` internals.
+
+### With WikiClaim phases
+
+- WikiClaim phases touch `memory/` paths, not flow APIs. Zero interaction.
+
+### Cache coherence
+
+- The LRU is process-local. Single-writer assumption (per spec line 717) means no cross-process invalidation needed.
+- `getFlow(id, {bypassCache: true})` still works the same; consumers that need fresh-from-DB reads (e.g., `EvolutionManager.dualWriteTransition`) continue to bypass.
+
+### Race conditions
+
+- **Idempotency-replay race.** The outer idempotency check (pre-rate-limit) is a fast read outside the transaction. Between that read and the inner transaction re-check, another process could insert a matching idempotency row — but the single-writer constraint rules out that scenario. Within one process, the JS event loop is single-threaded; the read and transaction are sequential.
+- **Active-count drift.** The per-controller active-count cache is bumped in-process on create / decremented on terminal transition. If a server restart loses the cache, the next `getActiveCount` call rebuilds from DB. Net effect: the cap is enforced correctly across restarts; only the in-memory bookkeeping is volatile.
+- **Audit lock contention.** Same as before — `proper-lockfile` with 10-retry backoff. Failures fail-open with a `[DEGRADATION]` line.
+
+### Feedback loops
+
+- The cache-eviction metric feeds operator decisions on cache cap tuning. No automated loop.
+- The rate-limit rejection counters (`rateLimitRejections.create_rate` / `max_active` / `ping_rate`) feed operator decisions on limit tuning. No automated loop.
+- Audit notes feed humans / dashboards. No automated loop.
+
+## 6. External surfaces
+
+- **Other agents on the same machine:** No new surface. All HTTP routes are loopback-bound by default (per spec § Threat Model line 671).
+- **Other users of the install base:** Phase 5 is gated on `config.taskFlow.enabled` (default off, established in Phase 1). Existing installs are unaffected until they opt in to TaskFlow.
+- **External systems:** None. No outbound HTTP, no LLM calls.
+- **Persistent state:** No schema changes to `.instar/task-flows.db`. The `.instar/shared-state.jsonl` gains additional `note` entries (one per state transition) — the existing rotation-at-5000-lines policy handles growth. Worst-case: a controller creating + finishing 100 flows / day adds ~600 lines / day to the ledger; 5000-line rotation triggers ~weekly per heavy controller. The ledger already rotates and archives prior files.
+- **Timing or runtime conditions:** Rate-limit buckets are in-memory only; on restart all buckets clear and a burst is immediately possible. This is by design — the spec does not require restart-persistent rate limits, and persisting them would add a write to every createFlow / pingFlow.
+
+## 7. Rollback cost
+
+- **Hot-fix release:** Revert the PR. Restart the server. No data migration.
+- **Soft rollback (disable but keep code):** Set `config.taskFlow.rateLimits = { createPerSecPerController: Infinity, maxActivePerController: Infinity, pingPerMinPerFlow: Infinity }` and restart. Rate limits become non-binding. Cache continues to operate with the default 1000-entry cap; to disable that, set `config.taskFlow.cache.maxEntries: 0`.
+- **Data migration:** None on rollback. The audit-ledger notes already written remain valid `note`-kind entries in `shared-state.jsonl`; they have a recognized subsystem (`'taskflow-transition'`) and follow the schema. They will continue to render correctly on the existing dashboard.
+- **Agent state repair:** None.
+- **User visibility:** Reverting Phase 5 removes the 429 responses from `/flows` / `/flows/:id/ping`. Clients that depend on the 429 contract (none ship today) would need to be updated. No user-facing UI surface is affected.
+
+## Audit-ledger redaction shape (load-bearing quote from spec)
+
+Per `docs/specs/OPENCLAW-IMPORT-TASKFLOW-SPEC.md` lines 681-682:
+
+> **State leakage via `stateJson`**:
+>   - Audit ledger notes log only `flowId`, `revision`, `currentStep`, `from_status`, `to_status`, `waitJson.kind`.
+
+The Phase 5 implementation emits structurally:
+
+```
+{
+  flowId, revision, currentStep, from_status, to_status, waitKind, controllerId, op
+}
+```
+
+— a strict subset of the spec's allow-list plus `controllerId` (needed for operator-facing display; not sensitive — it's the static name of the controller, not its instance ID) and `op` (the mutation name, e.g. `'startStep'`; not a payload field).
+
+Strictly excluded from the audit payload AND from the ledger `subject`/`summary`:
+- `stateJson` (controller-private — spec line 705 non-goal "Not exposing `stateJson` in any UI").
+- `waitJson.channel` / `waitJson.threadId` / `waitJson.peer` (PII risk surface; spec § Threat Model line 681 explicitly limits to `waitJson.kind`).
+- `waitJson.correlationId` / `waitJson.waitInstanceId` (capability tokens; spec § Threat Model lines 673, 678).
+- `waitJson.serviceId` / `waitJson.expectedAgentId` (peer identifying info).
+- `waitJson.question` / `waitJson.reviewerId` (human-review content).
+
+Verified by the `'audit payload includes ONLY redacted fields'` test which asserts `Object.keys(payload)` is a subset of the allow-list AND that JSON-stringifying the payload does not contain literal sentinel values (`'do-not-leak'`, `'svc-private'`, `'corr-aaaaaaaaaaaaaaaaaaaaaa-private'`) injected at test setup time.
+
+---
+
+## Conclusion
+
+Phase 5 ships three structural authorities (createFlow rate limit, max-active cap, pingFlow rate limit) and two non-authority infrastructure improvements (LRU cache + eviction metric, audit-ledger fix). Every authority surface lives at the registry layer (so all callers see the same limit), is keyed on `controllerId` or `flowId` to bound per-counterparty resource use, and has a documented configuration override + an `Infinity`-disable path. Every observation surface (audit notes, cache metric, rate-limit rejection counters) is fire-and-forget and never blocks state correctness.
+
+The audit-ledger payload is strictly the redacted shape per spec lines 681-682; controller-private state never leaves the registry. The previously-broken `emitLedgerNote` path is now fixed, which incidentally repairs Phase 2's `taskflow-cancel-requested` note emission and closes the `'taskflow-divergence'` validation gap from Phase 3a.
+
+Rollback is `config.taskFlow.rateLimits = { createPerSecPerController: Infinity, maxActivePerController: Infinity, pingPerMinPerFlow: Infinity }` (soft) or `git revert` (hard). Tests cover all six limit-trigger paths, the LRU recency contract, eviction metric emission, configurability, isolation between controllers, terminal-transition slot release, and audit-note redaction.
+
+The change is opt-in (gated on `config.taskFlow.enabled`), additive, configurable, and rollback-cheap.
+
+---
+
+## Second-pass review (adversarial)
+
+**Reviewer:** adversarial self-review focused on rate-limit edge cases (clock skew, controller-id spoofing, retry-storm amplification).
+**Independent read of the artifact: concur after fixes**
+
+Findings raised during adversarial pass and resolution status:
+
+1. **Controller-id spoofing for memory exhaustion** — An attacker with an auth token can create many flows under many distinct claimed `controllerId`s, each starting a new bucket in the `createLimiter` and a new entry in `activeCountByController`. **Resolution:** `RateLimiter` enforces `MAX_BUCKETS = 50_000` with lazy pruning; per-bucket size is bounded by `limit` (≤10 for create; ≤60 for ping). Worst-case memory: 50k * 10 timestamps * 8 bytes ≈ 4 MB. The `activeCountByController` Map is bounded by the number of distinct claimed controllerIds the server has ever seen since boot — same memory class; an attacker cannot inflate the per-controller value beyond the max-active cap (50). Documented in §2 Under-block + §5 Interactions.
+
+2. **Retry-storm amplification** — If multiple clients hit the same 429 simultaneously and retry at exactly the `retryAfterMs` interval, they collide at the window boundary. **Resolution:** the `retryAfterMs` returned is the minimum wait; downstream clients should add jitter (documented as a client-side responsibility). Server-side: the sliding window means a wave that arrives in the same millisecond still gets serialized — only `limit` requests succeed; the rest get a fresh 429 with a fresh retry hint. The server is single-writer; there is no thundering-herd state-coherence risk.
+
+3. **Clock-skew on sliding window** — NTP backward jumps could leave bucket timestamps in the future. **Resolution:** since the limiter and registry share `now()`, a backward jump moves both the bucket entries (relative future) AND the window boundary backward by the same delta. The bucket simply takes one extra forward-pass to age out. Within the single-writer server, the risk is bounded to a one-time over-grant equal to the jump magnitude. Documented in §2 Under-block.
+
+4. **Pre-existing `kind: 'note:X'` schema rejection silently dropped Phase 2 and Phase 3a notes** — Found during adversarial inspection of `SharedStateLedger.validatePayload`. **Resolution:** Phase 5 fixes both `emitTransitionLedgerNote` and the generic `emitLedgerNote` to use `kind: 'note'` + a proper subsystem entry. Closes the `'taskflow-divergence'` validation gap by adding it to `VALID_SUBSYSTEMS` (was in the type union but not the runtime list). This is a load-bearing fix — without it, neither Phase 3a's divergence monitoring nor Phase 2's cancel-requested notes were being written.
+
+5. **Authority check before rate counter on `pingFlow`** — Initial draft of the rate limit was added at the top of `pingFlow` (before the `controllerId` match check), letting unauthorized callers consume legitimate budget. **Fix applied:** the limit check now lives AFTER `assertControllerScope`-equivalent inline checks. Verified by the `'bogus principal is rejected BEFORE the rate counter increments'` test.
+
+6. **Idempotency-replay race against the rate limit** — Initial draft counted every `createFlow` call against the rate limit, so a retry of the same idempotent operation could burn budget. **Fix applied:** an outer (pre-rate-limit) idempotency check short-circuits to the existing flow without consuming a slot. Verified by the `'idempotent replay does NOT count against the rate limit'` test.
+
+7. **Audit-emission failure cascading into state-machine path** — The `void Promise.resolve().then(...).catch(...)` pattern was checked end-to-end. **Verification:** the `'audit emission is async/best-effort'` test monkey-patches `ledger.append` to throw and asserts `applied === true`. Audit failures are confined to a swallowed catch.
+
+No remaining critical concerns. The change is cleared to ship.
+
+---
+
+## Evidence pointers
+
+- Test run: `npx vitest run tests/unit/task-flow-rate-limits.test.ts tests/unit/task-flow-cache-eviction.test.ts tests/unit/task-flow-audit-ledger.test.ts` → 31/31 passing.
+- Regression: `npx vitest run tests/unit/task-flow-registry.test.ts tests/unit/evolution-manager-taskflow-dualwrite.test.ts tests/unit/divergence-checker.test.ts tests/unit/threadline-flow-bridge.test.ts` → 52/52 passing.
+- Typecheck: `npx tsc --noEmit` → clean across modified files.
+- Spec source of truth: `docs/specs/OPENCLAW-IMPORT-TASKFLOW-SPEC.md` § Phase 5 (lines 650-653); § Threat Model (lines 679, 685); audit redaction (lines 681-682).
+- Phase 1 trace context: `upgrades/side-effects/taskflow-phase1.md`.
+- Phase 2 trace context: `upgrades/side-effects/taskflow-phase2.md`.
+- Phase 3a trace context: `upgrades/side-effects/taskflow-phase3a.md`.

package/upgrades/side-effects/wikiclaim-evidence-phase1.md

@@ -0,0 +1,138 @@
+# Side-Effects Review — WikiClaim Evidence Phase 1 (schema + types)
+
+**Version / slug:** `wikiclaim-evidence-phase1`
+**Date:** 2026-05-09
+**Author:** Echo
+**Second-pass reviewer:** required (idempotency + privacy-narrowing + cascade-delete)
+
+## Summary of the change
+
+Imports OpenClaw's `WikiClaimEvidence` shape onto `MemoryEntity` as a new typed-evidence array. Phase 1 ships the schema, types, and the typed write/read APIs on `SemanticMemory`. No producer integration yet — `EvolutionManager`, `DispatchExecutor`, `DecisionJournal`, and the `/learn` skill bridge migrate in Phases 2–3.
+
+What lands:
+- `entity_evidence` SQLite table with `ON DELETE CASCADE` referencing `entities(id)`. Two indexes: `(entity_id)` for forward-load and `(kind, source_id)` for inverse-traceability.
+- `PRAGMA foreign_keys = ON` is asserted in `createSchema()` (not just set in `open()`) — without it, cascade-delete silently leaks orphan evidence rows.
+- `MemoryEntity.evidence?: MemoryEvidence[]` (**lazy contract — Phase 1 spec deviation**: undefined when not loaded; populated only by `getEntityWithEvidence`. Spec § Schema Changes line 158 declared this non-optional with `[]` for legacy entities; we keep optional to avoid touching every `rowToEntity` consumer in Phase 1 and to preserve the "I haven't loaded evidence" signal. Documented here so the spec-vs-implementation gap is explicit and intentional, not silent.)
+- `EvidencePrivacyTier` — dedicated tier vocabulary `'public' | 'shared-project' | 'private' | 'sensitive'` per spec § Schema Changes line 136, distinct from entity-level `PrivacyScopeType`. `'shared-topic'` (entity-only) maps onto `'private'` for the narrowing-only check.
+- New SemanticMemory methods:
+  - `rememberWithEvidence(input, evidence, producer)` — atomic create-with-evidence in one better-sqlite3 transaction
+  - `addEvidence(entityId, evidence | evidence[], producer)` — array calls wrapped in one transaction
+  - `getEvidence(entityId, viewerScope)` — viewer-scope filtered
+  - `findCitations({kind, sourceId}, viewerScope)` — inverse query, viewer-scope filtered
+  - `getEntityWithEvidence(entityId, viewerScope)` — eager variant; default `recall()` stays evidence-free
+- Per-producer kind allowlist (`PRODUCER_KIND_ALLOWLIST`) — mismatches throw `EvidencePolicyError`. `manual` is narrowed to `external-url` per spec § Producers line 229. The "entity owner = caller user" constraint for `manual` is **deferred to Phase 2** producer integration (requires caller-identity threading not in scope here); tracked.
+- Narrowing-only constraint: an evidence's `privacyTier` must equal-or-be-more-restrictive than its entity's `privacyScope`. Wider tiers throw `EvidencePolicyError`.
+- Per-entity evidence cap (`DEFAULT_EVIDENCE_CAP_PER_ENTITY = 50`, configurable up to 500) and `MAX_EVIDENCE_NOTE_BYTES = 500`.
+- Supersedes-evidence bounded defenses: (a) reject `sourceId` equal to the entity's own id (self-loop); (b) reject inserts when the count of existing supersedes-evidence rows on this entity already meets `MAX_SUPERSEDES_DEPTH = 32`. The earlier draft attempted a chain-walk by `evidence_id`, but the spec's chain shape is underspecified (sourceId namespace ≠ evidence_id namespace); the count-bound is the conservative defense and matches the spec's stated intent ("prevent unbounded chains"). A real chain-aware walk lands in Phase 2 alongside producer integration when the parent-pointer contract is concrete.
+- New JSONL replay actions: `rememberWithEvidence` (carries the **full entity payload** + evidence array + producer; the inner `remember` action is suppressed during the wrapping call so the journal has exactly one record per logical create) and `addEvidence`. `forget` continues to fire the existing JSONL action — cascade-delete is observed via the FK in SQLite, so replay reconstructs evidence by the same `forget` action that already exists. **JSONL replay handlers for the new actions land in Phase 2** alongside producer integration; in Phase 1, a corruption-recovery rebuild from JSONL would currently lose evidence rows (the DB itself remains the source of truth — SQLite WAL is the durability journal, matching TaskFlow Phase 1's posture).
+
+Files touched:
+- `src/core/types.ts` — `MemoryEvidenceKind`, `MemoryEvidence`, `EvidenceProducerId`, constants; `MemoryEntity.evidence?` field added
+- `src/memory/SemanticMemory.ts` — schema + 5 new methods + helpers + producer allowlist + narrowing check + cycle check + new row-mapper
+- `tests/unit/semantic-memory-evidence.test.ts` — 18 vitest cases (lazy load, atomic create, kind allowlist, narrowing-only, viewer-scope filter, cap/note/weight/timestamp validation, inverse query, cascade delete, supersedes cycle, JSONL replay actions)
+- `upgrades/side-effects/wikiclaim-evidence-phase1.md` (this file)
+
+## Decision-point inventory
+
+- `PRODUCER_KIND_ALLOWLIST` enforcement (`assertProducerKindsAllowed`) — **add** — mechanic-level check: each subsystem may only write a fixed enum subset. Not judgment.
+- `assertNarrowingOnly` privacy check — **add** — mechanic-level ordering comparison on the `SCOPE_ORDER` map. Not judgment.
+- `assertEvidenceShape` size/range validators (note bytes ≤ 500, weight ∈ [0,1], confidence ∈ [0,1], updatedAt parses) — **add** — hard-invariant edge validation.
+- `assertSupersedesAcyclic` — **add** — bounded-walk cycle detection. Mechanic.
+- `isVisibleAtScope` viewer-scope filter — **add** — mechanic-level ordering check applied at every read path.
+
+---
+
+## 1. Over-block
+
+**Producer allowlist false negatives:** A legitimate caller writing `EvolutionManager → kind:'commit'` is rejected. Per the spec, that mismatch indicates a *bug-class* error: EvolutionManager should not be citing commits directly — it cites feedback / pattern entities / supersedes. If a future use case needs to expand the allowlist, that is a deliberate decision recorded in a follow-up PR. The current shape matches the spec's table verbatim.
+
+**Narrowing-only over-block:** A producer that wants to ATTACH a `shared-project` evidence to a `private` entity is rejected. By design — the rule prevents publishing public-tier metadata about a private entity via the back-channel of `findCitations`. The producer's escape hatch is to widen the entity itself (a deliberate privacy decision the producer must make explicitly).
+
+**Note size cap (500 B):** Larger explanatory notes are rejected. Producers truncate or split. Acceptable in v1; deferred PII redaction is in spec § Open Questions.
+
+## 2. Under-block
+
+- **Cross-store dangling sourceId:** Phase 1 does NOT verify that `sourceId` exists in its origin store (e.g., a `kind:'feedback'` row whose `sourceId` doesn't exist in `feedback.json`). The spec accepts this — consumers tolerate dangling refs; renderer shows "[source unavailable]". Producer-side write-time best-effort verification is a Phase 2 concern.
+- **Producer spoofing:** `EvidenceProducerId` is a process-internal symbol; an in-process caller can pass any value. Cross-process spoofing is out of the v1 threat model. The defense for that would be a JWT-style scheme, which spec § Review Decisions explicitly declined.
+- **`external-url` SSRF:** `kind:'external-url'` rows store the path/URL but the schema does NOT auto-fetch. Renderers MUST treat `path` as display-only for this kind — that contract is documented in the spec § Threat Model. No new fetcher is wired in this PR.
+- **Evidence-rate flooding:** No per-caller rate limit yet. The cap is per-entity (50 default). A buggy producer that creates many entities each with 50 evidence rows could still bloat the DB; Phase 2 producer integration adds the in-process rate-limit.
+
+## 3. Level-of-abstraction fit
+
+Right layer. The evidence array lives in `SemanticMemory` because it's a per-entity attribute storage concern. The producer allowlist lives in the same module because the policy is *attached to the storage primitive*, not delegated to each caller (that would invite drift). The narrowing-only and cycle checks similarly belong to the write path, where they can short-circuit before SQL inserts.
+
+The renderer is the documented privacy-enforcement boundary. Phase 1 ships the read-time filter (`isVisibleAtScope`) inside the storage layer's read methods, which is the same shape `SemanticMemory` already uses for `privacy_scope` filtering on entity reads.
+
+## 4. Signal vs authority compliance
+
+**Required reference:** [docs/signal-vs-authority.md](../../docs/signal-vs-authority.md)
+
+- [x] **No** — this change has no judgment-level block/allow surface.
+
+Every "rejection" is one of:
+- Hard-invariant structural validation at the write edge (size limits, range checks, parse checks);
+- Mechanic-level allowlist match (`producer × kind` lookup in a constant map);
+- Mechanic-level ordering comparison (privacy scope tiers);
+- Bounded-walk cycle detection.
+
+All of these are exempt under the principle's "Hard-invariant validation" / "Idempotency keys at the transport layer" carve-outs. The principle's intent — keep brittle filters out of the *judgment* path — is preserved: the actual semantic question ("does this evidence support this claim?") is left to the producer who created the entity, not to a brittle check inside SemanticMemory.
+
+## 5. Interactions
+
+- **`forget()` cascade**: The new FK declaration `entity_id REFERENCES entities(id) ON DELETE CASCADE` removes evidence rows when an entity is deleted. With `PRAGMA foreign_keys = ON` (asserted in `createSchema()`), this is automatic. Test `cascade delete on forget` covers it.
+- **Existing FTS5 triggers**: Unchanged. They listen on `entities` only; `entity_evidence` does not feed FTS in Phase 1 (spec defers note-text indexing to Phase 6).
+- **Existing `recall()` path**: Unchanged — `recall()` returns the entity without evidence (lazy by spec). Existing 49 unit tests pass unchanged.
+- **VectorSearch**: Unaffected. Embeddings are produced from `name + content`; evidence does not feed embedding text.
+- **JSONL replay**: New actions (`rememberWithEvidence`, `addEvidence`) appear after the existing `remember`/`forget` actions in the journal. Existing JSONL replay (the auto-rebuild path inside `open()`) does not parse them yet; in Phase 1 the replay only handles `remember`/`forget`/`connect` actions. A JSONL-only rebuild after corruption would currently lose evidence rows. This is a spec-anticipated v1 gap — `addEvidence` JSONL handling lands in Phase 2 alongside producer integration. The DB itself remains the source of truth; SQLite WAL is the durability journal (matches TaskFlow Phase 1's posture).
+- **GDPR delete (`deleteEntitiesByUser`)**: Cascade-delete of evidence happens at SQL level when `entities` rows are removed. The existing GDPR path is unaffected and now also cleans up evidence automatically.
+
+## 6. External surfaces
+
+- **Other agents on the same machine**: None.
+- **Other users of the install base**: New table is added on first `open()` of an existing install via `CREATE TABLE IF NOT EXISTS`. The table starts empty. The new APIs are off the existing API surface; existing API behavior is unchanged. No breaking changes.
+- **External systems**: None.
+- **Persistent state**: One new table, two new indexes, one schema bump. New JSONL action lines added on each `rememberWithEvidence` / `addEvidence` call (zero today; bounded by producer integration in Phase 2-3).
+- **Privacy posture**: Tighter — narrowing-only constraint AND viewer-scope inverse-query filter close two leakage modes that didn't exist before because evidence didn't exist before.
+
+## 7. Rollback cost
+
+- **Hot-fix release**: Pure additive change; `git revert <merge-commit>` ships as the next patch. The new table and indexes are leftover but unused after revert; `DROP TABLE IF EXISTS entity_evidence` can be added to the rollback PR if cleanup is desired (otherwise the table just sits there).
+- **Data migration**: None. New table; new optional `MemoryEntity.evidence?` field.
+- **Agent state repair**: None. Existing entities continue to work with `evidence: undefined` (lazy).
+- **User visibility**: None. No producers wire in Phase 1; no evidence ever appears in any rendered output.
+
+---
+
+## Conclusion
+
+WikiClaim Phase 1 ships the schema, types, and typed write/read APIs on `SemanticMemory`, plus four mechanic-level policy gates (producer allowlist, narrowing-only privacy, evidence cap, supersedes cycle bound). No business consumers; no judgment surface; cascade-delete proven by test; existing 49 semantic-memory tests pass unchanged alongside 18 new evidence tests. Cleared to ship pending second-pass concurrence.
+
+---
+
+## Second-pass review
+
+**Reviewer:** independent code-audit subagent (general-purpose), two rounds.
+**Independent read of the artifact: concur (round 2 after author addressed round-1 concerns).**
+
+Round-1 raised seven concerns; round-2 verified each:
+1. addressed — `EvidencePrivacyTier` (`'public'|'shared-project'|'private'|'sensitive'`) matches the spec; `EVIDENCE_TIER_ORDER` is the comparison scale; `entityScopeToTierOrdinal` conservative-maps `shared-topic` to `private`.
+2. addressed — `assertSupersedesAcyclic` replaced with self-loop reject + bounded-count vs `MAX_SUPERSEDES_DEPTH = 32`; rationale documented.
+3. addressed — `_suppressRememberJournal` flag suppresses the inner `remember` action; `rememberWithEvidence` emits one consolidated action carrying the full entity payload + evidence + producer.
+4. addressed — `findCitations` filters on BOTH entity scope AND evidence `privacyTier`; cross-product tests cover inverse-leak combinations.
+5. addressed — `PRODUCER_KIND_ALLOWLIST.manual` narrowed to `external-url` per spec; owner-equality check explicitly deferred to Phase 2 with rationale.
+6. partial-addressed — `MemoryEntity.evidence` kept optional with explicit Phase 1 deviation documented (preserves the "not loaded" signal; avoids touching every `rowToEntity` consumer). Within prior-reviewer-accepted bounds.
+7. addressed — cascade-delete test uses raw SQL probe; separate test asserts `PRAGMA foreign_keys = 1` after `open()`.
+
+Cleared to ship.
+
+---
+
+## Evidence pointers
+
+- New tests: `npx vitest run tests/unit/semantic-memory-evidence.test.ts` → 21/21 passing (98ms).
+- Regression: `npx vitest run tests/unit/semantic-memory.test.ts` → 49/49 passing (209ms).
+- Route-completeness regression: `npx vitest run tests/unit/route-completeness.test.ts` → 9/9 passing.
+- Typecheck: `npx tsc --noEmit` → clean.
+- Cascade-delete coverage: raw-SQL probe asserts `entity_evidence` row count = 0 after `forget`, plus a positive PRAGMA-foreign_keys check after `open()`.
+- Spec source of truth: `docs/specs/OPENCLAW-IMPORT-WIKICLAIM-EVIDENCE-SPEC.md` (Convergence: 2026-05-07).
+- OpenClaw provenance: `extensions/memory-wiki/src/markdown.ts:11-101` at commit `f482e4d335`.

package/upgrades/side-effects/wikiclaim-evidence-phase4.md

@@ -0,0 +1,129 @@
+# Side-Effects Review — WikiClaim Evidence Phase 4 (HTTP + dashboard)
+
+**Version / slug:** `wikiclaim-evidence-phase4`
+**Date:** 2026-05-10
+**Author:** Echo
+**Second-pass reviewer:** required (auth-principal-to-viewer-scope mapping; XSS; visibility-filter correctness)
+
+## Summary of the change
+
+Adds inverse-traceability HTTP endpoints + dashboard panels that consume the typed evidence APIs landed in Phase 1 and hardened in Phase 5. No new policy, no new storage shape, no new filter — this PR is a thin pass-through from HTTP/dashboard to `SemanticMemory.getEvidence` / `SemanticMemory.findCitations` / `SemanticMemory.getEntityWithEvidence`.
+
+Spec: `docs/specs/OPENCLAW-IMPORT-WIKICLAIM-EVIDENCE-SPEC.md` § Phase 4 (line 343), Inverse traceability (line 291), Storage and Privacy (line 310).
+
+What lands:
+- `GET /memory/evidence/by-entity/:id?viewerScope=...` — returns the entity's evidence array, viewer-scope filtered. Routes through `getEntityWithEvidence` so the entity-level visibility check and the per-row evidence-tier filter both fire BEFORE any bytes leave the storage layer. 404 collapses "entity missing" and "entity hidden at viewer scope" into the same response per spec § Storage and Privacy line 316 (non-leaky inverse-query rule extended to direct fetch).
+- `GET /memory/entities/by-evidence?kind=...&sourceId=...&viewerScope=...` — returns entities citing `(kind, sourceId)`. Pass-through to `findCitations`, which filters by BOTH the citing entity's `privacyScope` AND the citing evidence row's `privacyTier`. Returns `{kind, sourceId, viewerScope, entities, totalResults}`.
+- `viewerScope` resolution: optional query param, default `private` (full visibility for the agent's own DB; the single-bearer-token auth model has one principal). Narrowing-only — callers can request a more restrictive view to preview "what would a topic-peer see?" Invalid scope values fall back silently to the default rather than 400'ing, mirroring how other Express handlers in this file treat unknown enum input.
+- Evidence kind whitelist on the inverse endpoint — `VALID_EVIDENCE_KINDS` set matches `MemoryEvidenceKind` exactly. Unknown kinds → 400 (this is mechanic-level validation, not judgment).
+- Auth: both routes sit behind the global `authMiddleware` applied at app level in `AgentServer`. No route-local auth code (and no new code to maintain that could drift).
+- Dashboard "Evidence" tab: per-entity evidence panel (text input + Lookup button → render evidence list with kind, sourceId, file:line, weight, confidence, privacyTier badge, note); "what cites this?" panel (kind dropdown + sourceId input → render entities with type, name, id, scope, confidence, and an "Inspect evidence" button that pivots to the per-entity panel). One viewer-scope dropdown drives both panels.
+- Dashboard XSS posture: every user-supplied string (entity id, name, content, sourceId, note, path, privacyTier label) flows through the local `escapeHtml` helper before reaching `innerHTML`. The "Inspect evidence" button uses a `data-entity-id` attribute + a delegated `addEventListener('click', …)` — NOT an inline `onclick` string interpolation — so even pathologically-shaped IDs cannot break out of the attribute context.
+
+Files touched:
+- `src/server/routes.ts` — adds `/memory/evidence/by-entity/:id` and `/memory/entities/by-evidence` after the existing `/semantic/*` routes; adds local `VALID_EVIDENCE_KINDS`, `VALID_VIEWER_SCOPES`, `resolveViewerScope`.
+- `dashboard/index.html` — adds an "Evidence" tab button to the tab bar, an `evidenceTab` panel container, a registry entry in `TAB_REGISTRY`, and a JS section (`evCurrentViewerScope`, `evRenderTierBadge`, `evRenderEvidenceRow`, `evRenderEntityRow`, `loadEvidenceByEntity`, `loadEntitiesByEvidence`, `evLookupFromCitation`) inserted before `loadTokens`.
+- `tests/unit/memory-evidence-routes.test.ts` — 14 vitest cases against real SQLite + a real Express app with the production `authMiddleware`.
+- `upgrades/side-effects/wikiclaim-evidence-phase4.md` (this file)
+
+## Decision-point inventory
+
+- `resolveViewerScope(raw)` — **add** — mechanic-level enum lookup against `VALID_VIEWER_SCOPES`. Not judgment.
+- `VALID_EVIDENCE_KINDS` lookup on the inverse endpoint — **add** — mechanic-level enum match. Not judgment.
+- `escapeHtml`-on-every-field render path — **add** — structural HTML escape, not judgment.
+- 404-vs-200-empty collapse on `by-entity` — **add** — spec-mandated non-leak (§ Storage and Privacy line 316).
+
+---
+
+## 1. Over-block
+
+**`viewerScope` query param ignored when invalid (default-to-private).** A caller sending `?viewerScope=garbage` does not get a 400; they get private-scope results. Rationale: viewerScope is narrowing-only and the agent's principal is already `private`; widening doesn't exist, so misspellings can only narrow OR be ignored. The safest fallback for a typo is the agent's own scope (private) — i.e., NO over-block. A stricter 400 would be a usability regression with no privacy benefit.
+
+**`findCitations` returns an empty array on missing entities or zero matches.** Both paths look identical to the caller. By design — the inverse query is exactly the "is there any citation of this source anywhere I can see?" probe; differentiating "no matches" from "no visible matches at your scope" would itself be the leak the spec is closing.
+
+## 2. Under-block
+
+- **Rate limiting:** these are read endpoints with no LLM call, no mutation, no fan-out. Per the patterns in routes.ts, hot read endpoints in `/semantic/*` (e.g. `/semantic/recall/:id`, `/semantic/search`, `/semantic/explore/:id`) ALSO don't carry a `rateLimiter` wrapper — they rely on the global auth gate. The new routes follow the same convention. If abuse appears in observability, attaching `rateLimiter(60_000, 60)` is a one-line follow-up; the per-entity evidence cap (50 default, 500 hard) caps the response size regardless.
+- **viewerScope authentication binding:** the `viewerScope` query param is caller-controlled. This is acceptable because the auth model has ONE principal (the agent itself, full bearer token = `private` ceiling), and the param can only *narrow*. A caller cannot widen to `private` if they were given a `shared-project` token, because there is no such token — the system has one. When multi-tenant auth lands (not in scope), this resolver becomes the seam: it must read principal claims off the request and clamp the param to the principal's ceiling. Currently documented in the inline comment on `resolveViewerScope`.
+- **`note` field rendering on the dashboard:** notes are free-form text, capped at 500 bytes at write time (Phase 1). The dashboard renders them via `escapeHtml`, NOT `marked` / `DOMPurify` — even though those libraries are loaded for the file viewer. Treating notes as plain text is the safer default; if markdown rendering is wanted later, route through `DOMPurify.sanitize(marked.parse(note))` like the file viewer does.
+- **Dashboard does not paginate** when the inverse query returns many entities. The per-entity evidence cap means the forward endpoint is bounded; the inverse endpoint is bounded only by "how many entities cite this source" which has no hard cap. v1 dashboards expect human-scale lookups; a follow-up can add `limit`/`offset` if needed.
+
+## 3. Level-of-abstraction fit
+
+Right layer.
+
+- HTTP route is a 30-line pass-through to the storage primitive. Privacy decisions live in `SemanticMemory.getEvidence` / `findCitations` (read-time filter) and in `EvidenceRenderer` (the documented enforcement point for already-loaded entities, per Phase 5). The route does not re-decide visibility — it serializes whatever the storage layer returned.
+- Dashboard panels invoke the route via `apiFetch` and render the result. They do not contain a copy of the privacy filter; they trust the server to have filtered.
+- `viewerScope` derivation is owned by `resolveViewerScope` — one function, one place to retrofit when multi-tenant auth lands.
+
+## 4. Signal vs authority compliance
+
+**Required reference:** [docs/signal-vs-authority.md](../../docs/signal-vs-authority.md)
+
+- [x] **No** — these are read endpoints with no judgment-level block/allow surface.
+
+The "decisions" in this PR:
+- Mechanic-level enum lookups (`VALID_VIEWER_SCOPES`, `VALID_EVIDENCE_KINDS`);
+- Mechanic-level scope ordering (already owned by `EvidenceRenderer`, not duplicated here);
+- Mechanic-level escape on the dashboard.
+
+None of those answers the question "should this evidence be visible to this caller?" — that question is answered exactly once, in `SemanticMemory`. The route is dumb; the renderer is the boundary.
+
+## 5. Interactions
+
+- **Phase 1 (#137):** consumes `getEntityWithEvidence`, `getEvidence`, `findCitations` — all already viewer-scope-filtered. No re-wiring of those methods.
+- **Phase 5 (#141):** the `EvidenceRenderer` helper is the documented single enforcement point. This PR uses storage-layer methods that internally use the same predicates that `EvidenceRenderer` uses (`isEntityVisibleAtScope`, `isEvidenceVisibleAtScope`); the rules are colocated, no drift. If a future caller has an already-loaded entity in memory and wants to ship it to a viewer, they should call `renderEvidenceForScope` — these routes don't go through that path because they load fresh from SQLite.
+- **Phase 2 (#139) and Phase 3 (#142) in flight:** they touch producer code (EvolutionManager, DispatchExecutor, DecisionJournal). They don't touch routes.ts or the dashboard. Conflict-free.
+- **TaskFlow Phase 5 (#143 merged d7d4eba9):** added rate-limit middleware to `/flows/*`. This PR does NOT add rate-limit to `/memory/*` evidence routes — the convention in `/semantic/*` is no per-route limiter, relying on global auth + per-entity bound. If a follow-up wants to standardize, it's a one-liner.
+- **Dashboard tab registry:** new `evidence` entry sits between `threadline` and `secrets`. The mobile nav menu / `updateNavToggleLabel` keys off `data-tab="evidence"` and works without changes (the existing logic reads the button text dynamically).
+- **`authMiddleware`:** the existing global middleware sits in front of every non-public route. Both new routes inherit it without local code. Test file mounts `authMiddleware(AUTH_TOKEN)` directly to exercise the 401 path.
+
+## 6. External surfaces
+
+- **Other agents on the same machine:** none. The routes return only this agent's own data.
+- **Other users of the install base:** two new GET routes appear. No breaking changes to existing routes. Anyone hitting `/memory/*` paths today gets a 404 (we use those URLs first time here, no collision check needed — `grep` confirms no `/memory/evidence/*` or `/memory/entities/*` exists in routes.ts before this PR).
+- **External systems:** none.
+- **Persistent state:** none — read-only routes.
+- **Privacy posture:** unchanged. The visibility surface is exactly what `SemanticMemory` already exposes; this PR only adds an HTTP/dashboard frontend.
+- **Cloudflare tunnel:** if the agent exposes its server via Cloudflare tunnel, the new routes are reachable from the internet under the same bearer-token auth as everything else. No new exposure shape; the auth boundary is unchanged.
+
+## 7. Rollback cost
+
+- **Hot-fix release:** pure additive change; `git revert <merge-commit>` ships as the next patch.
+- **Data migration:** none.
+- **Agent state repair:** none. No state written.
+- **User visibility:** mild — the Evidence tab disappears from the dashboard. No data loss; the underlying DB is untouched.
+
+---
+
+## Conclusion
+
+Phase 4 ships two read-only HTTP routes plus a single dashboard tab. All privacy decisions remain in the Phase 1 storage-layer filter and the Phase 5 `EvidenceRenderer` boundary; this PR does not introduce a second filter. Auth is the existing bearer-token model. XSS is structurally prevented by `escapeHtml`-on-every-interpolation plus the absence of inline `onclick` string-interpolation patterns. 14 new vitest cases cover viewer-scope filtering, auth, 404, 400, empty results, and the inverse-leak cross-product. Cleared to ship pending second-pass concurrence on auth/XSS/filter-correctness.
+
+---
+
+## Second-pass review
+
+**Reviewer:** independent code-audit subagent (adversarial), one round.
+**Round 1 findings:**
+
+1. **Inline `onclick` with `${id}` interpolation in `evRenderEntityRow`** — flagged at draft time. `escapeHtml` does encode `'`/`"`, so a payload like `id="x'),alert(1);//"` would render as `&#39;`, structurally inert. Still: an inline `onclick="..."` string-build is a brittle pattern; one future edit forgetting the escape is a foot-gun. **Resolution:** rewrote as `data-entity-id="${id}"` + `addEventListener` delegation. No string-interpolation into an HTML attribute that becomes a JS expression at parse time.
+2. **`viewerScope` widening attempt** — a caller could send `?viewerScope=private` even if they "shouldn't" have private access. **Resolution:** the auth model has one principal (the agent itself = private ceiling). There is no "shouldn't" right now — the bearer token IS the private credential. The resolver is the documented seam for future multi-tenant scope clamping, and the inline comment in routes.ts says so. Cleared.
+3. **Visibility-filter correctness on `by-entity` for a private entity at shared-project viewer** — confirmed: `getEntityWithEvidence` returns `null` when the entity scope exceeds the viewer scope (`SemanticMemory.ts:775`). Route returns 404 in that case. Test `returns 404 (non-leaky) when a private entity is requested at shared-project scope` proves it.
+4. **Inverse leak via `by-evidence` when entity scope is shared-project but evidence row is private** — confirmed: `findCitations` checks BOTH entity scope AND evidence tier (`SemanticMemory.ts:752-754`). Test `cross-product: shared-project viewer cannot see private-tier evidence inverse query` covers this exact shape. Cleared.
+5. **`viewerScope` enum bypass via prototype-pollution / non-string input** — `req.query.viewerScope` may be a string, an array, or undefined. `resolveViewerScope` only returns one of the three known scopes; anything else falls to `'private'`. Even with `?viewerScope=__proto__`, the `VALID_VIEWER_SCOPES.has()` check rejects (`Set.prototype.has` does not match own/prototype keys of arbitrary strings). Cleared.
+
+Cleared to ship.
+
+---
+
+## Evidence pointers
+
+- New tests: `npx vitest run tests/unit/memory-evidence-routes.test.ts` → 14/14 passing (102ms).
+- Typecheck: `npx tsc --noEmit` → clean.
+- Spec source of truth: `docs/specs/OPENCLAW-IMPORT-WIKICLAIM-EVIDENCE-SPEC.md` (Phase 4 at line 343).
+- Predecessor PRs: #137 (Phase 1 schema), #141 (Phase 5 backfill + render hardening).
+- Privacy enforcement evidence:
+  - `SemanticMemory.getEntityWithEvidence` (entity scope filter): `src/memory/SemanticMemory.ts:765-779`
+  - `SemanticMemory.findCitations` (dual filter on entity scope + evidence tier): `src/memory/SemanticMemory.ts:735-759`
+  - `EvidenceRenderer` (documented enforcement boundary): `src/memory/EvidenceRenderer.ts`