memhouse 0.20.1 → 0.20.2

package/CHANGELOG.md

@@ -4,6 +4,67 @@ Versions before 0.8.0 were beta-only. Beta installs of 0.7.x and earlier should
 uninstall and reinstall — the 0.8.0 layout is new, and an in-place `update` +
 migration path is planned work, not a promise the old versions can cash.
 
+## 0.20.2 — 2026-11-03
+
+Release notes: [`docs/releases/v0.20.2.md`](docs/releases/v0.20.2.md). Read its "Read this
+before upgrading".
+
+- **Before upgrading: the first capture re-derives every session from its archive.** memhouse now
+  reads editors through `@agent-realm/harness-session-adapters` 1.1.0, whose adapters state
+  `adapter_ver` 0.3.0. A new adapter version is an upgrade (harness-sessions-sql §8, "Re-derive from
+  the archive"), so the first pass after `memhouse update` replays each session's archived passes
+  under 0.3.0 and writes its rows again above the house's revs. It reads the archive, not the
+  editors' stores, and writes every session once; on a large house that first pass takes longer
+  than a usual one. Later passes are as before. The editors memhouse reads through its own session
+  logs (every editor but Claude Code and Codex) re-derive in the same pass: their adapter,
+  `memhouse-bridge`, moves to 1.0.2 for the Cursor sub-agent links (below).
+- **The profile tables follow harness-sessions-sql v1.1: a record's native fields, and the units it
+  came from.** Five columns, added to an existing house by the schema step (`ADD COLUMN`, at the
+  end of the table; a new house has them at the specification's positions):
+  - `sessions.native` and `session_parts.message_native`, `part_native`: the fields of the
+    editor's own record that the adapter declares (a Claude Code message's `message.stop_reason`,
+    say), as canonical JSON, `''` when there are none;
+  - `sessions.units` and `session_parts.message_units`: the `(source_key, generation, unit_key)`
+    of each archived unit the record came from, so a row joins back to its raw line in
+    `<db>_archive.<m>_session_units`.
+  A row the capture wrote before 0.20.2 holds `''` and `[]` until its session is re-derived (above).
+  The legacy conversion leaves the five columns out of its `content_sha` line, as v1.1 asks, so a
+  converted session keeps its shas.
+- **A member's reading view of turns: `<m>_session_turns_view`** (harness-sessions-sql v1.1 §6,
+  `session_turns`, still a plan there). One row per turn of one session, a turn running from a
+  prompt to the next one, with its messages, tool calls, failed tool results and the models that
+  served it:
+  `SELECT * FROM <db>.<m>_session_turns_view(host = '…', harness = '…', harness_path = '…', session_id = '…', incarnation = '')`.
+  The schema step makes it (`CREATE OR REPLACE`). It runs as its reader (`SQL SECURITY INVOKER`),
+  so it shows nobody a row the tables would not, and a share does not reach it: a share is still
+  the six tables. It is not named `<m>_session_turns`, because a table ending in `_turns` reads as
+  a 0.18 member's room when memhouse looks for colliding member names.
+
+- **`memhouse room-sql` runs `SHOW ROW POLICIES` on ClickHouse 25.11.** It appended its output
+  format (`FORMAT PrettyCompact`, or `JSONEachRow` with `--json`) to the statement, and 25.11 parses
+  no FORMAT after a bare `SHOW ROW POLICIES` or `SHOW POLICIES`, so the statement failed with a
+  syntax error. The format now goes as the `default_format` setting, which applies to every
+  statement; a FORMAT the statement names itself still wins.
+- **The development layout (`<m>_turns`, `MEMHOUSE_LAYOUT=v3-dev`): an adapter upgrade writes above
+  the rows it replaces.** A new adapter version re-derived an unchanged file, from disk or from the
+  archive, at the rev the file's last pass had written, so its rows tied with the old ones and won
+  only by insertion order (listed under Deferred in 0.20.0). It now writes one rev above the
+  highest the house holds for that file, tombstones included. No derive code changed, so nothing is
+  re-derived. The profile tables were not affected.
+- **Cursor: a chat's sub-agents are captured as its links.** Cursor names a chat's sub-agents only
+  in the chat's own record (`subagentComposerIds`), never in a message, and the profile capture did
+  not read them there, so the links from a chat to its sub-agents were missing. The log adapter
+  (`memhouse-bridge` 1.0.2) now reads them as links stated on the chat, one per sub-agent. No live
+  Cursor store has shown a sub-agent yet: the tests use synthetic sessions. Each new sub-agent
+  still rewrites the chat's log; placing it in the log where it appeared waits for a live sample
+  (`docs/design/sessions-engine.md`).
+- **A session log memhouse cannot read is never rewritten into an empty one.** memhouse refuses to
+  rewrite the normalised log of a session that had messages into one with none, since an empty
+  read is far likelier a misread store than a real clear. It counted the old log's messages as 0
+  when the log was there but could not be read (a permission or disk error), which let that
+  rewrite through. Such a log now stops the session's pass with "cannot read the existing log
+  (<code>)", reported and retried like any other failed read.
+
 ## 0.20.1 — 2026-11-02
 
 Release notes: [`docs/releases/v0.20.1.md`](docs/releases/v0.20.1.md).

package/bin/memhouse.js

@@ -2896,14 +2896,13 @@ async function cmdRoomSql() {
     const body0 = qc().rewriteRooms(sql, { member, db: cfg.db, schema });
     qc().assertRead(body0);
     if (flags.print === true) { console.log(body0); return 0; }
-    const stripped = body0.trim().replace(/;\s*$/, '');
+    // The format travels as a setting, never appended to the statement (q-cli roomSqlRequest).
+    const req = qc().roomSqlRequest(body0, { json: JSON_OUT });
+    const text = await t.raw(req.body, { database: cfg.db, settings: req.settings });
     if (JSON_OUT) {
-      if (qc().hasFormat(stripped)) return qError(Object.assign(new Error('--json picks the format itself — drop the FORMAT clause'), { status: 400 }));
-      const text = await t.raw(`${stripped}\nFORMAT JSONEachRow`, { database: cfg.db });
       console.log(JSON.stringify(text.split('\n').filter(Boolean).map((l) => JSON.parse(l))));
       return 0;
     }
-    const text = await t.raw(qc().hasFormat(stripped) ? stripped : `${stripped}\nFORMAT PrettyCompact`, { database: cfg.db });
     process.stdout.write(text.endsWith('\n') || !text ? text : `${text}\n`);
     return 0;
   } catch (e) { return qError(e); }

package/docs/guides/share.md

@@ -26,6 +26,7 @@ Full or scoped, a share is `SELECT` on exactly your six conversation tables:
 | rows | all | those matching the scope, through a row policy on each table |
 | capture archive (`<db>_archive`) | not granted | not granted |
 | `<you>_meta`, `<you>_events` | not granted | not granted |
+| `<you>_session_turns_view` (your turns, a view) | not granted | not granted |
 | tables added later | not granted | not granted |
 
 ## Scopes

package/docs/releases/README.md

@@ -5,6 +5,7 @@ do, how to upgrade, what is deprecated and when it goes, and what was deferred.
 
 | version | date | headline |
 |---|---|---|
+| [0.20.2](v0.20.2.md) | 2026-11-03 | harness-sessions-sql v1.1's columns and a turns view; Cursor sub-agent links; the first capture re-derives every session |
 | [0.20.1](v0.20.1.md) | 2026-11-02 | invites expire; `memhouse plugins install codex`; per-command help and clearer CLI answers |
 | [0.20.0](v0.20.0.md) | 2026-10-29 | schema 3: a session is its store, nothing cut or lost, forks billed once, ten editors through normalisers; Apache-2.0 |
 | [0.18.11](v0.18.11.md) | 2026-10-10 | sessions with subagents stop re-shipping; security: no stored credential sent to `invite --url` |

package/docs/releases/v0.20.2.md

@@ -0,0 +1,134 @@
+# memhouse 0.20.2 — 2026-11-03
+
+A patch release on 0.20.1. The profile tables gain five columns, which the schema step adds in
+place, and the first capture after upgrading re-derives every session from its archive. Install
+it with `memhouse update`, or `npm install -g memhouse`.
+
+- **The profile tables gain harness-sessions-sql v1.1's five columns.** Each session and message
+  keeps the editor's own fields that its adapter declares, and the archived units it came from,
+  so a row joins back to its raw line. These notes claim no v1.1 conformance; the conformance
+  run's result is below.
+- **A reading view of turns:** `<m>_session_turns_view`, one row per turn of a session (v1.1's
+  `session_turns`, still a plan there).
+- **Cursor: a chat's sub-agents are captured as its links.**
+- **Fixes:**
+  - `memhouse room-sql` runs `SHOW ROW POLICIES` on ClickHouse 25.11;
+  - a session log memhouse cannot read is never rewritten into an empty one;
+  - on the development layout, an adapter upgrade writes above the rows it replaces.
+
+## Read this before upgrading
+
+- **The first capture re-derives every session from its archive.** This happens once, on the
+  first pass after `memhouse update`. It reads the archive, not the editors' stores, and writes
+  every session's rows again, so it takes longer than a usual pass; later passes are as before.
+  - Measured on a synthetic house of 2,500 sessions (2,000 Claude Code and 500 Codex; 110 MiB
+    of session files, 193,814 message parts), in a 4-core container with 6 GiB of memory, that
+    pass took 18 minutes on ClickHouse 26.7.4.58 and 27 minutes on 25.11.9.34: 1.5 and 1.3
+    times as long as capturing the same house for the first time. The shipper's memory peaked
+    at 348 MB and ClickHouse's at 58 MiB per query, and the next pass took 7 seconds.
+  - The time grows with the number of sessions, 0.43 seconds a session on 26.7 and 0.64 on
+    25.11 in that measurement: a house with four times as many sessions takes about four times
+    as long.
+  - Claude Code and Codex are read through `@agent-realm/harness-session-adapters` 1.1.0, whose
+    adapters state `adapter_ver` 0.3.0.
+  - Every other editor is read through memhouse's own session logs, whose adapter,
+    `memhouse-bridge`, moves to 1.0.2 (the Cursor links below).
+  - A new adapter version is an upgrade (harness-sessions-sql §8, "Re-derive from the
+    archive"), so each session is replayed under the new version and written above the house's
+    revs.
+- **The schema step adds five columns** to an existing house's `<m>_sessions` and
+  `<m>_session_parts`, at the end of each table (a new house has them at the specification's
+  positions). The member's own credential adds them; the archive tables do not change. A row
+  captured before 0.20.2 holds `''` and `[]` there until its session is re-derived, which the
+  first capture does.
+
+```bash
+memhouse update        # then: memhouse status — every line a check mark
+```
+
+## The profile tables: harness-sessions-sql v1.1's columns
+
+This adds the v1.1 columns and the `session_turns` view (a plan in v1.1).
+
+- `sessions.native`, `session_parts.message_native` and `part_native`: the fields of the
+  editor's own record that the adapter declares (a Claude Code message's `message.stop_reason`,
+  say), as canonical JSON, `''` when there are none.
+- `sessions.units` and `session_parts.message_units`: the `(source_key, generation, unit_key)`
+  of each archived unit the record came from, so a row joins back to its raw line in
+  `<db>_archive.<m>_session_units`.
+- The legacy conversion leaves the five columns out of its `content_sha` line, as v1.1 asks, so
+  a converted session keeps its shas.
+
+These notes claim no conformance. The harness-sessions-sql conformance kit, run against this
+release's code on ClickHouse 26.7 and 25.11 with the session adapters 1.1.0:
+
+- **Native is declared** passes.
+- **Units resolve** fails. memhouse numbers a source's first archived generation 0, where v1.1
+  numbers it 1, so each triple's generation is one lower than the specification's. Every triple
+  still names a unit of the session's own archive, so a row joins back to its raw line. memhouse
+  has numbered generations this way since 0.20.0. Aligning the two is 0.20.3 work.
+- **Crash-safe** keeps the failure 0.20.0 reported: a stopped read whose source changed before
+  the re-run.
+
+## A reading view of turns
+
+`<m>_session_turns_view` (harness-sessions-sql v1.1 §6, `session_turns`, still a plan there)
+gives one row per turn of one session, a turn running from a prompt to the next one, with its
+messages, tool calls, failed tool results and the models that served it:
+
+```sql
+SELECT * FROM <db>.<m>_session_turns_view(host = '…', harness = '…', harness_path = '…', session_id = '…', incarnation = '')
+```
+
+- The schema step makes it (`CREATE OR REPLACE`).
+- It runs as its reader (`SQL SECURITY INVOKER`), so it shows nobody a row the tables would not.
+- A share does not reach it: a share is still the six tables.
+- It is not named `<m>_session_turns`, because a table ending in `_turns` reads as a 0.18
+  member's room when memhouse looks for colliding member names.
+
+## Cursor sub-agents
+
+Cursor names a chat's sub-agents only in the chat's own record (`subagentComposerIds`), never in
+a message, and the profile capture did not read them there, so the links from a chat to its
+sub-agents were missing. `memhouse-bridge` 1.0.2 reads them as links stated on the chat, one per
+sub-agent.
+
+- No live Cursor store has shown a sub-agent yet; the tests use synthetic sessions.
+- Each new sub-agent still rewrites the chat's normalised log. Placing it where it appeared
+  waits for a live sample (Deferred).
+
+## Smaller changes
+
+- **`memhouse room-sql` runs `SHOW ROW POLICIES` on ClickHouse 25.11.** It appended its output
+  format to the statement, and 25.11 parses no FORMAT after a bare `SHOW ROW POLICIES` or `SHOW
+  POLICIES`. The format now goes as the `default_format` setting; a FORMAT the statement names
+  itself still wins.
+- **A session log memhouse cannot read is never rewritten into an empty one.** An old log that
+  was there but could not be read (a permission or disk error) counted as having no messages,
+  which let an empty read through. It now stops that session's pass with "cannot read the
+  existing log (<code>)", reported and retried.
+- **The development layout (`<m>_turns`, `MEMHOUSE_LAYOUT=v3-dev`):** an adapter upgrade writes
+  one rev above the highest the house holds for the file, instead of tying with the rows it
+  replaces. The profile tables were not affected.
+- **The site** no longer points readers at a repository-only document.
+
+## Deferred
+
+- **The normalisers' filesystem errors** (0.20.3): about thirty reads in the editor normalisers
+  take any error for "not there", so a permission denial (macOS privacy on
+  `~/Library/Application Support`, for one) shows as no sessions. 0.20.3 reports them by editor
+  and errno in `status` and `doctor`, with a Full Disk Access hint on macOS.
+- **Generation numbering** (0.20.3): a source's first archived generation is 0 where v1.1 says 1
+  (above).
+- **Cursor sub-agents as appended records.** An anchor would place each after the parent's last
+  message at or before the sub-agent's own `createdAt`; that waits for a live store to confirm
+  that a composer's and its bubbles' `createdAt` share one clock.
+- **`text_len` on the development layout** counts UTF-16 units, not code points.
+- **`memhouse reset` and `relocate` on schema 3**: 0.21, with a design note first.
+
+## Deprecated
+
+Unchanged from 0.20.0: passwords on the command line (`--password`, `--member-password`,
+`--to-password`), and `MEMHOUSE_USER` / `MEMHOUSE_PASSWORD` as the admin for `migrate --v3-*`,
+still work with a warning and are refused from the next minor release. See
+[0.20.0's Deprecated](v0.20.0.md#deprecated).

package/memhouse/house/q-cli.js

@@ -741,6 +741,19 @@ function hasFormat(sql) {
   return /\bFORMAT\s+[A-Za-z0-9_]+\s*$/i.test(s);
 }
 
+/**
+ * What `memhouse room-sql` sends: the statement as written, and the output format as the
+ * default_format setting. A FORMAT appended to the statement fails where the statement takes
+ * none (ClickHouse 25.11 parses no FORMAT after a bare SHOW ROW POLICIES); a setting applies to
+ * every statement, and a FORMAT the statement names itself still wins. --json picks the format,
+ * so a statement that names one is refused there.
+ */
+function roomSqlRequest(sql, { json = false } = {}) {
+  const body = String(sql).trim().replace(/;\s*$/, '');
+  if (json && hasFormat(body)) throw new QueryInputError('--json picks the format itself — drop the FORMAT clause');
+  return { body, settings: { default_format: json ? 'JSONEachRow' : 'PrettyCompact' } };
+}
+
 /** Refuse what is not a read before it leaves the machine (the server's readonly=2 refuses it too). */
 function assertRead(sql) {
   const s = String(sql).replace(/--[^\n]*$/gm, '').replace(/\/\*[\s\S]*?\*\//g, '').trim();
@@ -858,6 +871,6 @@ module.exports = {
   renderSchemaBlocks,
   NAMES, ALIASES, FILTER_FLAGS, QueryInputError,
   schemaOf, httpTransport, fmtTop, fmtInner, parseWhen, filtersFrom, parseRef, parseQ, runQ, renderQ, allQFlags, canonicalName,
-  resolveRef, hitSpec, lookupSpec, withIso, rewriteRooms, hasFormat, assertRead, statsV3, renderStats, statusV3, statusSpec, machinesFolded,
+  resolveRef, hitSpec, lookupSpec, withIso, rewriteRooms, hasFormat, roomSqlRequest, assertRead, statsV3, renderStats, statusV3, statusSpec, machinesFolded,
   sessionsQueryV3, resumeSpec, table,
 };

package/memhouse/house/records-to-profile.js

@@ -9,6 +9,12 @@
 // record without content_sha and cmp_sha) on every row; on a part row, the sha of the part object
 // as it sits in its message's parts, with message_content_sha the message's. A message's header
 // row (part_id '', kind 'none') carries the message's sha in both.
+//
+// native and units (harness-sessions-sql v1.1, sessions-model §2.7): a record's `native` as
+// canonical JSON text, its `units` as (source_key, generation, unit_key) tuples. A part row carries
+// its message's (message_native, message_units) and its own part_native. A record without them
+// (an adapter older than sessions-model v1.4) leaves the columns out, so the server's defaults
+// ('' and []) stand and a row mapped before them keeps its digest.
 
 const crypto = require('crypto');
 
@@ -30,6 +36,10 @@ const str = (v) => (v == null ? '' : String(v));
 const nb = (v) => (typeof v === 'boolean' ? v : null);
 const nn = (v) => (Number.isFinite(Number(v)) && v !== null && v !== '' ? Number(v) : null);
 const tsOrNull = (v) => (v && Number.isFinite(Date.parse(v)) ? v : null);
+/** A record's native as its column holds it: canonical JSON, or undefined (the column's default) when absent. */
+const nativeText = (v) => (v != null && typeof v === 'object' ? jcs(v) : undefined);
+/** A record's units as named tuples, or undefined (the column's default) when the record states none. */
+const unitsOf = (v) => (Array.isArray(v) ? v.map(([sourceKey, generation, unitKey]) => ({ source_key: str(sourceKey), generation: Number(generation) || 0, unit_key: str(unitKey) })) : undefined);
 
 /** The part columns of one part. */
 function partColumns(p) {
@@ -39,7 +49,7 @@ function partColumns(p) {
     call_id: '', tool_name: '', args: '', server: '', is_error: null, result_status: '', output: '', output_ref: '', output_ref_sha256: '', output_ref_bytes: null,
     output_len: null, truncated: null, exit_code: null, media_type: '', inline_b64: '', ref: '', bytes_len: null, opaque: null, signature_len: null,
     replaces_form: '', replaces_value: '', error_kind: '', error_message: '', http_status: null, level: '', native_type: '',
-    part_source: str(p.source), part_state: str(p.state),
+    part_source: str(p.source), part_state: str(p.state), part_native: nativeText(p.native),
   };
   switch (p.kind) {
     case 'text': case 'notice': case 'summary': case 'reasoning':
@@ -99,7 +109,8 @@ function profileRows(records, key, { firstSeen = () => '1970-01-01T00:00:00.000Z
       source: str(m.source), to_user: nb(m.to_user), to_model: nb(m.to_model), status: str(m.status), state: str(m.state), on_active_branch: nb(m.on_active_branch),
       ts, requested_model: str(m.requested_model), served_model: str(m.served_model), effort: str(m.effort), response_id: str(m.response_id),
       message_aliases: Array.isArray(m.aliases) ? m.aliases.map((a) => ({ id: str(a.id), kind: str(a.kind) })) : [],
-      cmp_sha: str(m.cmp_sha), message_content_sha: msgSha, first_seen: firstSeen(str(m.message_id)),
+      cmp_sha: str(m.cmp_sha), message_content_sha: msgSha, message_native: nativeText(m.native), message_units: unitsOf(m.units),
+      first_seen: firstSeen(str(m.message_id)),
     };
     const ps = Array.isArray(m.parts) ? m.parts : [];
     ps.forEach((p, i) => {
@@ -159,7 +170,7 @@ function profileRows(records, key, { firstSeen = () => '1970-01-01T00:00:00.000Z
     mode: str(s.mode), kind: str(s.kind), reuse: str(s.reuse), ts_source: str(s.ts_source),
     aliases: Array.isArray(s.aliases) ? s.aliases.map((a) => ({ id: str(a.id), kind: str(a.kind) })) : [],
     fingerprint: s.fingerprint && Array.isArray(s.fingerprint.cmp_shas) ? s.fingerprint.cmp_shas.map(String) : [],
-    setup: s.setup == null ? null : jcs(s.setup), content_sha: recordSha(s),
+    setup: s.setup == null ? null : jcs(s.setup), native: nativeText(s.native), units: unitsOf(s.units), content_sha: recordSha(s),
   } : null;
   return { parts, usage, links, lifecycle, session, calls: [...calls.values()].filter((c) => c.call_id && c.tool_name !== undefined) };
 }

package/memhouse/house/schema-profile-views.sql.tpl

@@ -0,0 +1,53 @@
+-- memhouse's reading views over one member's profile tables: harness-sessions-sql v1.1 §6
+-- (schema/clickhouse-views.sql), each reading the member's own {db}.{m}_* tables. Rendered by
+-- schema-profile.js with {db} and {m}, after the tables (schema-profile.sql.tpl). Views are not
+-- tables: they are never in the column heal or the key check.
+--   * CREATE OR REPLACE, so a house created before a definition changed gets the new one at its
+--     next ensure (install, update, migrate).
+--   * SQL SECURITY INVOKER: a read runs with the reader's own grants and row policies (share.js),
+--     never the creator's, so a view shows a reader no row the tables would not.
+--   * {m}_session_turns_view, not {m}_session_turns: a table name ending in _turns reads as a 0.18
+--     member's turns room (share.js memberCollisions).
+
+-- §6, session_turns (plan): one row per turn of one session. A turn starts at a message a person
+-- sent in the main conversation (role 'user', source 'user', agent_id '') and runs to the next
+-- one; messages before the first prompt are turn 0. The session key is the view's parameters, so
+-- the filter on the sort-key prefix comes before FINAL:
+--   SELECT * FROM {db}.{m}_session_turns_view(host = '…', harness = '…', harness_path = '…',
+--                                              session_id = '…', incarnation = '')
+CREATE OR REPLACE VIEW {db}.{m}_session_turns_view SQL SECURITY INVOKER AS
+WITH
+  parts AS (
+    SELECT * FROM {db}.{m}_session_parts FINAL
+    WHERE host = {host:String} AND harness = {harness:String}
+      AND harness_path = {harness_path:String}
+      AND session_id = {session_id:String} AND incarnation = {incarnation:String}
+      AND is_deleted = 0 AND removed = 0
+      AND state = 'current' AND part_state IN ('', 'current')
+      AND ifNull(on_active_branch, true)
+  ),
+  messages AS (
+    SELECT message_id,
+           min(order_key) AS order_key,
+           toUInt32(any(role) = 'user' AND any(source) = 'user' AND any(agent_id) = '') AS is_prompt
+    FROM parts
+    GROUP BY message_id
+  ),
+  numbered AS (
+    SELECT message_id, is_prompt,
+           sum(is_prompt) OVER (ORDER BY order_key, message_id
+                                ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS turn
+    FROM messages
+  )
+SELECT n.turn                                                AS turn,
+       anyIf(p.message_id, n.is_prompt = 1)                  AS prompt_message_id,
+       min(p.ts)                                             AS started_at,
+       max(p.ts)                                             AS ended_at,
+       uniqExact(p.message_id)                               AS messages,
+       countIf(p.kind = 'tool_call')                         AS tool_calls,
+       countIf(p.kind = 'tool_result' AND ifNull(p.is_error, false)) AS tool_errors,
+       arraySort(groupUniqArrayIf(p.served_model, p.served_model != '')) AS served_models
+FROM parts AS p
+INNER JOIN numbered AS n USING (message_id)
+GROUP BY n.turn
+ORDER BY n.turn;

package/memhouse/house/schema-profile.js

@@ -8,7 +8,10 @@
 //                                       option, nothing in <db>_archive granted to anyone else
 //
 // The DDL lives in schema-profile.sql.tpl, rendered with {db} and {m}; every statement is
-// CREATE … IF NOT EXISTS. Transport-free like schema-v3.js: every entry point takes `q`.
+// CREATE … IF NOT EXISTS. The reading views (harness-sessions-sql v1.1 §6) live apart, in
+// schema-profile-views.sql.tpl, as CREATE OR REPLACE VIEW: they are not tables, so they stay out
+// of templateColumns, the column heal and the key check. Transport-free like schema-v3.js: every
+// entry point takes `q`.
 
 const fs = require('fs');
 const path = require('path');
@@ -16,6 +19,7 @@ const { assertUsableName, legacyTextIndexDialect, isTextIndexGrammarRefusal, tex
 const { stripComments, qFromClient, DDL_SETTINGS } = require('./schema-v3');
 
 const TEMPLATE = path.join(__dirname, 'schema-profile.sql.tpl');
+const VIEWS_TEMPLATE = path.join(__dirname, 'schema-profile-views.sql.tpl');
 
 // The columns retyped in harness-sessions-sql v1, which ensureProfileSchema brings an older table to:
 // adapter_ver UInt32 -> LowCardinality(String), child_stored Bool -> Nullable(Bool), orig_sha256
@@ -47,19 +51,43 @@ const SERVER_COLUMNS = ['writer', 'ingested_at', 'search_ngram', 'search_word',
 
 function archiveDb(db) { return `${db}_archive`; }
 
-/** Every fully-qualified name of one member's profile tables. */
+/** The reading views, by their name after `<m>_` (schema-profile-views.sql.tpl). */
+const VIEW_TYPES = ['session_turns_view'];
+
+/** Every fully-qualified name of one member's profile tables and views. */
 function profileNames(db, member) {
   assertUsableName(db, 'house');
   assertUsableName(member, 'member');
   const out = { db, archive: archiveDb(db), member };
   for (const t of CONVERSATION_TYPES) out[t] = `${db}.${member}_${t}`;
   for (const t of CAPTURE_TYPES) out[t] = `${archiveDb(db)}.${member}_${t}`;
+  for (const t of VIEW_TYPES) out[t] = `${db}.${member}_${t}`;
   out.meta = `${db}.${member}_meta`;
   out.events = `${db}.${member}_events`;
   return out;
 }
 
 function readTemplate() { return fs.readFileSync(TEMPLATE, 'utf-8'); }
+function readViewsTemplate() { return fs.readFileSync(VIEWS_TEMPLATE, 'utf-8'); }
+
+/** The view statements for one member, in template order: [{ type, fq, sql }]. Each reads the member's own house. */
+function renderViews(tpl, { db, member }) {
+  assertUsableName(db, 'house');
+  assertUsableName(member, 'member');
+  const body = stripComments(tpl).replaceAll('{db}', db).replaceAll('{m}', member);
+  const out = [];
+  for (const raw of body.split(';')) {
+    const sql = raw.trim();
+    if (!sql) continue;
+    const m = /^CREATE OR REPLACE VIEW ([A-Za-z0-9_]+)\.([A-Za-z0-9_]+) SQL SECURITY INVOKER AS\b/.exec(sql);
+    if (!m) throw new Error(`schema-profile views template: unexpected statement: ${sql.slice(0, 80)}`);
+    if (m[1] !== db) throw new Error(`schema-profile views template: '${m[1]}.${m[2]}' is outside the house`);
+    const type = m[2].startsWith(`${member}_`) ? m[2].slice(member.length + 1) : m[2];
+    if (!VIEW_TYPES.includes(type)) throw new Error(`schema-profile views template declares an unknown view '${m[2]}'`);
+    out.push({ type, fq: `${m[1]}.${m[2]}`, sql });
+  }
+  return out;
+}
 
 /** The CREATE statements for one member, in template order: [{ type, database, table, fq, sql }]. */
 function renderStatements(tpl, { db, member }) {
@@ -156,12 +184,12 @@ function msg(e) { return e && e.message ? e.message : String(e); }
  * Create one member's profile tables, idempotently, then verify them against PROFILE_KEYS.
  * As ensureV3Schema: a privilege denial is counted and skipped, a lost create race is success.
  */
-async function ensureProfileSchema(q, { db, member, createDatabases = false, grantCapture = false, tpl = null } = {}, ui = null) {
+async function ensureProfileSchema(q, { db, member, createDatabases = false, grantCapture = false, tpl = null, viewsTpl = null } = {}, ui = null) {
   const n = profileNames(db, member);
   const say = (m) => ui && ui.ok && ui.ok(m);
   const version = String(((await q.rows('SELECT version() AS v', { final: 0 }))[0] || {}).v || '');
   const dialect = textIndexDialectFor(version);
-  const out = { version, dialect, created: [], denied: [], unhealed: [], grants: [], grant_failures: [], tables: {}, ok: false };
+  const out = { version, dialect, created: [], denied: [], unhealed: [], grants: [], grant_failures: [], views: [], view_failures: [], tables: {}, ok: false };
   if (createDatabases) {
     for (const d of [n.db, n.archive]) {
       try { await q.sql(`CREATE DATABASE IF NOT EXISTS ${d}`, DDL_SETTINGS); } catch (e) {
@@ -257,6 +285,18 @@ async function ensureProfileSchema(q, { db, member, createDatabases = false, gra
     }
   }
   if (out.retyped.length) say(`profile tables for ${member}: retyped ${out.retyped.join(', ')}`);
+  // The reading views, over session_parts as healed above. A view is a reader's convenience, not
+  // something the capture writes: a refusal goes to `denied`, any other failure to view_failures,
+  // and neither makes the tables not ok. Not over a session_parts that is missing or lacks a column.
+  if (out.created.includes(n.session_parts) && !lacking.has('session_parts')) {
+    for (const v of renderViews(viewsTpl || readViewsTemplate(), { db, member })) {
+      try { await q.sql(v.sql, DDL_SETTINGS); out.views.push(v.fq); } catch (e) {
+        const m = msg(e);
+        if (isDenied(m)) out.denied.push(`CREATE VIEW ${v.fq}`); else out.view_failures.push(`${v.fq}: ${m.split('\n')[0]}`);
+      }
+    }
+    if (out.views.length) say(`reading views for ${member}: ${out.views.join(', ')}`);
+  }
   if (grantCapture) {
     for (const g of captureGrants(db, member)) {
       try { await q.sql(g, DDL_SETTINGS); out.grants.push(g); } catch (e) {
@@ -284,7 +324,7 @@ async function ensureProfileSchema(q, { db, member, createDatabases = false, gra
 }
 
 module.exports = {
-  TEMPLATE, CONVERSATION_TYPES, CAPTURE_TYPES, PROFILE_TYPES, PROFILE_KEYS, SERVER_COLUMNS,
-  archiveDb, profileNames, readTemplate, renderStatements, templateColumns, templateKeys, rowColumns, insertColumns,
+  TEMPLATE, VIEWS_TEMPLATE, CONVERSATION_TYPES, CAPTURE_TYPES, PROFILE_TYPES, VIEW_TYPES, PROFILE_KEYS, SERVER_COLUMNS,
+  archiveDb, profileNames, readTemplate, readViewsTemplate, renderStatements, renderViews, templateColumns, templateKeys, rowColumns, insertColumns,
   keyProblem, captureGrants, ensureProfileSchema, qFromClient,
 };

package/memhouse/house/schema-profile.sql.tpl

@@ -1,5 +1,7 @@
 -- memhouse's house tables as harness-sessions-sql's ClickHouse profile (step B1, tables and keys). The
--- tables, columns, keys and engines are harness-sessions-sql's schema/clickhouse.sql (v1.0.0);
+-- tables, columns, keys and engines are harness-sessions-sql's schema/clickhouse.sql (v1.1: v1's,
+-- with the native and units columns of §4.2 and §4.3; its reading view is
+-- schema-profile-views.sql.tpl);
 -- memhouse adds, beside them:
 --   * the member prefix {m}_ and the archive database {db}_archive (spec §3: physical names);
 --   * on session_parts: first_seen (a first-seen time, never written into ts), folder, project,
@@ -48,6 +50,8 @@ CREATE TABLE IF NOT EXISTS {db}.{m}_session_parts (
     message_aliases      Array(Tuple(id String, kind LowCardinality(String))),
     cmp_sha              FixedString(64),
     message_content_sha  FixedString(64),
+    message_native       String DEFAULT '',       -- v1.1: the message's native fields (sessions-model §2.7), canonical JSON; '' when none
+    message_units        Array(Tuple(source_key String, generation UInt32, unit_key String)),  -- v1.1: the units the message came from
     kind                 LowCardinality(String),
     text                 String DEFAULT '' CODEC(ZSTD(3)),
     call_id              String DEFAULT '',
@@ -78,6 +82,7 @@ CREATE TABLE IF NOT EXISTS {db}.{m}_session_parts (
     native_type          LowCardinality(String) DEFAULT '',
     part_source          LowCardinality(String) DEFAULT '',
     part_state           LowCardinality(String) DEFAULT '',
+    part_native          String DEFAULT '',       -- v1.1: the part's own native fields, canonical JSON; '' when none
     content_sha          FixedString(64),
     adapter              LowCardinality(String),
     adapter_ver          LowCardinality(String),
@@ -131,6 +136,8 @@ CREATE TABLE IF NOT EXISTS {db}.{m}_sessions (
     aliases          Array(Tuple(id String, kind LowCardinality(String))),
     fingerprint      Array(FixedString(64)),
     setup            Nullable(String),
+    native           String DEFAULT '',                -- v1.1: the session's native fields (sessions-model §2.7), canonical JSON; '' when none
+    units            Array(Tuple(source_key String, generation UInt32, unit_key String)),  -- v1.1: the units the session record came from
     content_sha      FixedString(64),
     adapter          LowCardinality(String),
     adapter_ver      LowCardinality(String),

package/memhouse/share.js

@@ -195,9 +195,11 @@ function memberCollisions(member, tables) {
 }
 
 // The names memhouse gives a member's tables in the house, as the part after `<m>_`: its tables,
-// the copies migrate keeps of a 0.18 room while it moves one (`<room>__migrating`) and after
-// (`<room>_pre_epoch`), and the scratch table share's probe makes and drops (`_mh_probe_<pid>`).
-const MEMBER_TABLES = [...new Set([...V3_TABLES, ...LEGACY_ROOMS, ...DEV_ROOMS, 'meta', 'events', 'session_stats', 'session_model_stats', 'session_tool_stats',
+// its reading views (schema-profile.js VIEW_TYPES: session_turns_view, named so it does not end in
+// _turns, which memberCollisions reads as a 0.18 member's room), the copies migrate keeps of a
+// 0.18 room while it moves one (`<room>__migrating`) and after (`<room>_pre_epoch`), and the
+// scratch table share's probe makes and drops (`_mh_probe_<pid>`).
+const MEMBER_TABLES = [...new Set([...V3_TABLES, ...LEGACY_ROOMS, ...DEV_ROOMS, 'meta', 'events', 'session_stats', 'session_model_stats', 'session_tool_stats', 'session_turns_view',
   ...LEGACY_ROOMS.flatMap((r) => [`${r}_pre_epoch`, `${r}__migrating`])])];
 const isMemberTable = (rest) => MEMBER_TABLES.includes(rest) || /^_mh_probe_\d+$/.test(rest);
 const startsMemberTable = (p) => [...MEMBER_TABLES, '_mh_probe_1'].some((n) => n.startsWith(`${p}_`));