create-walle 0.9.28 → 0.9.30

package/template/claude-task-manager/docs/backfill-incremental-no-main-fallback.md

@@ -0,0 +1,48 @@
+# apiSessionMessages: no-main-fallback + incremental low-priority backfill
+
+Three rules from the user:
+1. **Never fall back to the main loop.** Remove the `onMain()` heavy compute from the
+   session-message off-thread builders. On read-pool failure/degrade → defer (serve stale
+   or `503 {deferred}` retry), never project/serialize on main.
+2. **Incremental fetch/import.** Backfill a single (possibly giant) session in bounded
+   row-chunks across ticks, resumable, instead of one all-or-nothing `replaceSessionMessageRows`.
+3. **Smaller chunks, lower priority.** Backfill sweep: smaller per-tick session limit + wall
+   budget + per-session row cap, and SKIP the tick under a transient load surge so interactive
+   reads keep the read pool / write lock.
+
+## Root cause (from live perf logs)
+The one-time migration backfill saturates the read pool (`session-content-rows-backfill`
+caught blocking 6107ms; a single 16k-message session is migrated in ONE
+`replaceSessionMessageRows` call → tens of seconds holding the write lock). While the pool is
+saturated, `apiSessionMessages` off-thread reads time out and fall back to `onMain()` → the
+full-conversation projection + `JSON.stringify` runs on the main loop → 1.5–2.8s freezes
+(probe-confirmed: `apiSessionMessages:2781ms`, `serializeResponse`×20, `project`×37).
+
+## Changes
+
+### db.js
+- `getSessionMessageRowCount(sessionId)` — dense-store resume point.
+- `appendSessionMessageRowsChunk(sessionId, messages, startIndex, maxCount)` — `INSERT OR IGNORE`
+  rows for indices `[start, min(start+maxCount, len))` (idempotent, indices preserved), and set
+  `extracted_source_len` only when the chunk reaches the end. Returns `{written, nextIndex, complete}`.
+
+### lib/session-content-backfill.js
+- `runContentRowsBackfillSweep` gains chunked mode: when `opts.appendChunk`/`opts.getRowCount`
+  /`opts.maxRowsPerSession` are provided, migrate each session one bounded chunk per tick
+  (resume from existing row count); a still-incomplete session marks `partial=true` so the
+  reclaim/legacy-drop passes wait. Falls back to whole-session `replaceSessionMessageRows`
+  when chunk deps are absent (back-compat for existing tests).
+
+### server.js
+- Rule 1: `_buildSessionMessagesFullResponseOffThread`, `_buildSessionMessagesPageResponseOffThread`,
+  `_fetchSessionMessagesPageOffThread` → `onFail`/degrade returns `null` (no `onMain()`).
+  `_sendFullMessagesOffThread` on `null` serves the per-session last-good buffer if present,
+  else `503 {deferred:true}` with `Retry-After`. Never runs projection/serialize on main.
+- Rule 3: backfill sweep registration uses smaller `limit` + `budgetMs` + `maxRowsPerSession`
+  and skips the tick when `_underTransientLoadSurge()` (or worker degraded, already present).
+
+## Tests
+- `tests/session-content-backfill-chunked.test.js` — chunked backfill: ≤ maxRows/session/tick,
+  resumes from existing count, converges to 100%, idempotent, gate stays false until complete.
+- `tests/session-messages-no-main-fallback.test.js` — the off-thread builders return null on
+  pool-fail; the full-read handler serves stale/503 and never invokes the on-main projection.

package/template/claude-task-manager/docs/conversation-import-freshness.md

@@ -109,3 +109,24 @@ For a stale Conversation tab, compare:
 
 If the provider JSONL is current but the DB cache is stale, the fix belongs in
 the importer/scheduler path, not in xterm rendering or tab switching.
+
+## Conversation Read Contract
+
+The Conversation tab is a CTM DB read path. Provider JSONL files are the
+authoritative import/repair source, but they must not be parsed on the normal
+`/api/session/messages?mode=turns&compact=conversation` request path.
+
+Request behavior:
+
+- Serve `session_message_rows` when rows exist, even if the source JSONL has
+  grown past the imported DB watermark.
+- When rows are present but stale, return the DB page with
+  `cacheFresh:false`, `stale:true`, and `freshness:"catching_up"`, then wake
+  the importer out of band.
+- When rows are absent but `session_conversations` has cached messages, serve
+  that DB cache and wake import if stale or explicitly refreshed.
+- When neither rows nor cache exist, return an explicit DB cache miss response
+  and wake import instead of parsing JSONL inside the HTTP request.
+
+The importer is responsible for keeping CTM DB current. The HTTP request is
+responsible for bounded DB reads and honest freshness metadata.