memhouse 0.3.2 → 0.4.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +64 -6
- package/bin/memhouse.js +882 -40
- package/index.js +2 -2
- package/mem-house/COMPETITION.md +217 -0
- package/mem-house/DESIGN.md +61 -32
- package/mem-house/delivery/AGENT-INSTALL.md +5 -3
- package/mem-house/delivery/PROMPT.md +23 -10
- package/mem-house/delivery/install.sh +9 -7
- package/mem-house/delivery/kernel-install.md +56 -25
- package/mem-house/delivery/skills/memhouse-search/SKILL.md +46 -16
- package/mem-house/delivery/skills/memhouse-sessions/SKILL.md +38 -7
- package/mem-house/delivery/skills/memhouse-sql/SKILL.md +48 -17
- package/mem-house/deploy.js +333 -0
- package/mem-house/envfile.js +49 -0
- package/mem-house/per-member/INSTALL.md +183 -0
- package/mem-house/per-member/PLAN.md +174 -0
- package/mem-house/per-member/PROVISIONING.md +177 -0
- package/mem-house/per-member/README.md +48 -0
- package/mem-house/per-member/SCHEMA.md +73 -0
- package/mem-house/per-member/SHARING.md +102 -0
- package/mem-house/per-member/provision.js +142 -0
- package/mem-house/per-member/rooms.js +132 -0
- package/mem-house/per-member/schema-member.sql.tpl +85 -0
- package/mem-house/per-member/schema-merge.sql.tpl +24 -0
- package/mem-house/server/queries.js +103 -53
- package/mem-house/service.js +400 -0
- package/mem-house/shipper/ship.js +61 -18
- package/package.json +3 -3
- package/pricing.js +20 -2
- package/pricing.json +216 -24
- package/public/assets/{index-CCPh7a2Z.js → index-CtH-5i95.js} +5 -5
- package/public/index.html +1 -1
- package/ui/src/App.jsx +3 -2
- package/ui/src/pages/SqlViewer.jsx +39 -21
- package/mem-house/rls.sql +0 -17
- package/mem-house/schema.sql +0 -145
|
@@ -0,0 +1,174 @@
|
|
|
1
|
+
# per-member rooms
|
|
2
|
+
**v6 reverts one decision v5 made under review pressure.** v5 gave every member a stored
|
|
3
|
+
`sessions_v`. SCHEMA v4 had already decided that the rollup is a *saved query*, and
|
|
4
|
+
overriding that reactively — inside a review round, to fix a read-layer finding — cost a
|
|
5
|
+
name inside the Merge selector's namespace (161 sessions counted as 322), a rename, a
|
|
6
|
+
reserved prefix, a fourth grant and a fourth object to provision and roll forward. The
|
|
7
|
+
query is back. SCHEMA v5 records the detour in full.
|
|
8
|
+
|
|
9
|
+
**v5 corrects the grant set, adds the member's own view, and stops calling this a
|
|
10
|
+
skeleton.** Four changes:
|
|
11
|
+
|
|
12
|
+
1. The grant set was `SELECT, INSERT`, which is insufficient — the shipper clears a
|
|
13
|
+
session's rows before re-inserting, so every incremental pass needs both
|
|
14
|
+
`ALTER UPDATE` and `ALTER DELETE` (which of the two is demanded varies by server
|
|
15
|
+
version). The error surfaced only on the *second* ship of a *changed* session, which is
|
|
16
|
+
why four document versions carried it. See PROVISIONING v5.
|
|
17
|
+
2. The delete predicate **must bind the user** rather than call `currentUser()`. A mutation
|
|
18
|
+
does not evaluate it in the caller's context, so the delete matched nothing and removed
|
|
19
|
+
nothing, silently. Measured on 25.11 and on chdb; it affected the shared layout too.
|
|
20
|
+
3. The read layer resolves the rollup as **SQL text** rather than a name: the stored view
|
|
21
|
+
in the shared layout, a subquery over the caller's rooms in the per-member one. Without
|
|
22
|
+
that resolution a per-member house ships fine and reads back nothing.
|
|
23
|
+
4. v1–v4 were design skeletons. This one is implemented and measured: `rooms.js`,
|
|
24
|
+
`provision.js`, the two DDL templates, and room routing through the read layer.
|
|
25
|
+
|
|
26
|
+
## The model
|
|
27
|
+
|
|
28
|
+
House **`mem`**. Three rooms per member:
|
|
29
|
+
|
|
30
|
+
```
|
|
31
|
+
mem.sessions_<member>
|
|
32
|
+
mem.messages_<member>
|
|
33
|
+
mem.tool_calls_<member>
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
Owner-managed Merge rooms for team-wide reads:
|
|
37
|
+
|
|
38
|
+
```sql
|
|
39
|
+
mem.all_sessions = Merge('mem', '^sessions_')
|
|
40
|
+
mem.all_messages = Merge('mem', '^messages_')
|
|
41
|
+
mem.all_tool_calls = Merge('mem', '^tool_calls_')
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Canon: house = database, room = table, member = a registered user with grants.
|
|
45
|
+
|
|
46
|
+
## Why
|
|
47
|
+
|
|
48
|
+
**Isolation is structural.** A grant that is absent exposes nothing; a row policy has to
|
|
49
|
+
be correct on every room for every role, and one wrong predicate exposes everything. It
|
|
50
|
+
fails closed.
|
|
51
|
+
|
|
52
|
+
**Sharing is self-serve.** A member holding grant-option on their own rooms shares with a
|
|
53
|
+
colleague directly. Under row policies the same act needs `CREATE ROW POLICY`, which the
|
|
54
|
+
kernel withholds from agency owners.
|
|
55
|
+
|
|
56
|
+
**Personal reads stop paying for everyone else.** The shared rooms order
|
|
57
|
+
`(session_id, user_id, seq)`, so filtering by member cannot prefix-scan. At a hundred
|
|
58
|
+
members every personal query and full-text lookup scans roughly 100× what it needs.
|
|
59
|
+
|
|
60
|
+
**Per-member lifecycle is atomic** — delete, export, retention, and quota are per-room
|
|
61
|
+
operations rather than mutations across a shared table.
|
|
62
|
+
|
|
63
|
+
## Grants
|
|
64
|
+
|
|
65
|
+
Two statements per room, six per member, explicit — plus SELECT on the Merge rooms.
|
|
66
|
+
`PROVISIONING.md` is the authority; this is the shape:
|
|
67
|
+
|
|
68
|
+
```sql
|
|
69
|
+
GRANT ALL ON mem.sessions_<m> TO <m>;
|
|
70
|
+
GRANT SELECT ON mem.sessions_<m> TO <m> WITH GRANT OPTION;
|
|
71
|
+
-- and the same pair for messages_<m> and tool_calls_<m>
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
**The split is the point.** `ALL` covers the mutation privileges the shipper's re-ship
|
|
75
|
+
needs without naming them — clear-then-insert, so a growing transcript cannot leave a
|
|
76
|
+
stale `seq` tail when a re-parse yields fewer rows, and which of `ALTER UPDATE` /
|
|
77
|
+
`ALTER DELETE` a server demands varies by version. Grant-option is attached to the
|
|
78
|
+
`SELECT` statement and **not** to the `ALL` one, so a share is read-only by construction:
|
|
79
|
+
a member cannot hand a colleague `DROP TABLE` while meaning "let them read my sessions".
|
|
80
|
+
Never `ON mem.*` — see PROVISIONING.
|
|
81
|
+
|
|
82
|
+
## What stays
|
|
83
|
+
|
|
84
|
+
- `user_id MATERIALIZED currentUser()` on every room; `async_insert=0` on writers.
|
|
85
|
+
- Typed columns, `extra JSON`, `ReplacingMergeTree(ingested_at)`, `final=1` reads, both
|
|
86
|
+
full-text layers on messages.
|
|
87
|
+
- Parse-on-client; the shipper runs the 17 adapters and writes typed rows.
|
|
88
|
+
- `sessions_v` as a **saved query** run under the caller, not a stored view — the v1–v4
|
|
89
|
+
decision, restored. The read layer resolves it as text in the same place it resolves a
|
|
90
|
+
room name, so the rollup and the rooms travel through one substitution point.
|
|
91
|
+
- The REST contract and the borrowed dashboard.
|
|
92
|
+
|
|
93
|
+
## Measured facts (ClickHouse 26.7.1)
|
|
94
|
+
|
|
95
|
+
1. Members **self-provision** inside their own prefix and are denied outside it.
|
|
96
|
+
2. `Merge` **fails closed on permissions** — a caller sees only the underlying rooms they
|
|
97
|
+
hold grants for, with no leak and no error. This requires SELECT on the Merge room
|
|
98
|
+
ITSELF: without it the team room denies outright instead of narrowing, which is a
|
|
99
|
+
different and worse behaviour. Provisioning grants it (re-measured on 25.11).
|
|
100
|
+
3. `Merge` **auto-discovers** rooms created after it exists.
|
|
101
|
+
4. `Merge` **tolerates schema drift**, returning a column's default for rooms lacking it —
|
|
102
|
+
so schema rollout is progressive, not lock-step. It **rejects mutations**; use
|
|
103
|
+
`ADD COLUMN`, Merge room first.
|
|
104
|
+
5. Row-policy rights are **house-scopeable** (`GRANT ACCESS MANAGEMENT ON mem.*`). Owner
|
|
105
|
+
only — the scope is the house, not a room.
|
|
106
|
+
6. **`EXECUTE AS` exists**, changes `currentUser()`, and is a grantable privilege denied to
|
|
107
|
+
members by default.
|
|
108
|
+
7. **The shipper needs both mutation grants** on its own rooms, or incremental passes
|
|
109
|
+
fail — 26.7 named `ALTER DELETE`, 25.11 named `ALTER UPDATE(_row_exists)`. New in v5.
|
|
110
|
+
8. **`currentUser()` in a mutation predicate matches nothing.** A `DELETE` filtered on
|
|
111
|
+
`user_id = currentUser()` removed 0 rows where the same predicate with the bound value
|
|
112
|
+
removed 2000 — on ClickHouse 25.11 and on chdb alike, with no error either time. New
|
|
113
|
+
in v5.
|
|
114
|
+
|
|
115
|
+
## Measured end-to-end (the implementation, not the design)
|
|
116
|
+
|
|
117
|
+
On ClickHouse 25.11, two members, real local session stores:
|
|
118
|
+
|
|
119
|
+
- alice ships 161 sessions / 22,413 message rows / 16,197 tool rows into her own rooms;
|
|
120
|
+
bob's rooms are untouched.
|
|
121
|
+
- alice reading **or writing** bob's rooms is denied, and so is reading bob's view.
|
|
122
|
+
- the whole read layer resolves: 13 of 14 dashboard query functions run green against
|
|
123
|
+
per-member rooms (the fourteenth needs a SQL argument), and `status`, `search`, `doctor`,
|
|
124
|
+
`stats` all report alice's own data. The same queries against the shared names fail with
|
|
125
|
+
`UNKNOWN_TABLE`, which is what they did before this change.
|
|
126
|
+
- a planted stale `seq` tail is cleared by the next re-ship, which is what the two ALTER
|
|
127
|
+
grants exist for.
|
|
128
|
+
- the whole gauntlet scenario's SQL assertions pass when run by hand against a two-member
|
|
129
|
+
house: 22,413 rows in alice's room, none unstamped, none in bob's, bob reading the team
|
|
130
|
+
Merge room sees zero of alice's, alice sees her own. Sharing round-trips: alice's
|
|
131
|
+
`GRANT` widens bob's Merge view to 22,413 and her `REVOKE` returns it to zero, with the
|
|
132
|
+
direct read denied again.
|
|
133
|
+
- the shared-room layout has been removed; this is the only layout.
|
|
134
|
+
|
|
135
|
+
## `user_id` — what it guarantees
|
|
136
|
+
|
|
137
|
+
Unforgeable **by a member**, since members do not hold `EXECUTE AS`. Forgeable by the
|
|
138
|
+
owner or any grantee. The trust boundary is the owner, not the engine — the normal
|
|
139
|
+
arrangement, but do not overstate it anywhere user-facing. **Members must never hold
|
|
140
|
+
`EXECUTE AS`.**
|
|
141
|
+
|
|
142
|
+
## Phases
|
|
143
|
+
|
|
144
|
+
1. Schema + naming — room DDL template, key order, Merge rooms. **Done.**
|
|
145
|
+
2. Provisioning — kernel capability and standalone path. **Standalone done**
|
|
146
|
+
(`provision.js`); the kernel capability is not written.
|
|
147
|
+
3. Query layer — the member view and room-name resolution. **Done** — `rooms.js` plus
|
|
148
|
+
templated room names through `queries.js`, `ship.js` and the CLI.
|
|
149
|
+
4. Sharing — room-level self-serve; row-level via the owner. **Room-level proven**;
|
|
150
|
+
row-level not implemented.
|
|
151
|
+
|
|
152
|
+
Migration is not a phase and must not shape MVP decisions.
|
|
153
|
+
|
|
154
|
+
## Open questions
|
|
155
|
+
|
|
156
|
+
- **May members create and drop their own rooms?** Works, safely scoped; but a member
|
|
157
|
+
could drop their own memory and schema rollout stops being central. Deferred with the
|
|
158
|
+
uninstall/drop work.
|
|
159
|
+
- **Key order** — is `user_id` still worth an `ORDER BY` slot now the room is the tenant?
|
|
160
|
+
- **Standalone path** — a user with no kernel is both owner and member; one credential or
|
|
161
|
+
two?
|
|
162
|
+
- **Is `EXECUTE AS` scopeable** to `mem.*` rather than `*.*`? Bounds what an owner
|
|
163
|
+
credential can impersonate.
|
|
164
|
+
- ~~**Should a share be `SELECT`-only?**~~ **RESOLVED.** Grant-option is attached to the
|
|
165
|
+
member's `SELECT` grant and not to their `ALL` grant, so a share can only ever be
|
|
166
|
+
read-only. Measured on 26.7.2: `GRANT SELECT … TO bob` succeeds,
|
|
167
|
+
`GRANT DROP TABLE … TO bob` is refused 497.
|
|
168
|
+
|
|
169
|
+
## Out of scope
|
|
170
|
+
|
|
171
|
+
- Migration. Deferred; unblocked in principle by `EXECUTE AS`.
|
|
172
|
+
- `agent_id` / crew identity. Within a crew sharing one credential an agent tag is
|
|
173
|
+
client-supplied and is not a security boundary.
|
|
174
|
+
- Adapters, REST contract, dashboard.
|
|
@@ -0,0 +1,177 @@
|
|
|
1
|
+
# PROVISIONING — minting a member and their rooms
|
|
2
|
+
|
|
3
|
+
Two statements per room, six per member, and no seventh for the rollup: the session
|
|
4
|
+
rollup is a saved query over exactly these rooms, so a member who can read them can run
|
|
5
|
+
it. See [INSTALL.md](INSTALL.md) for the three ways this gets applied.
|
|
6
|
+
|
|
7
|
+
## A note on names before the SQL
|
|
8
|
+
|
|
9
|
+
The house is `mem`. `provision.js`, the shipper, the read layer and the CLI all default to
|
|
10
|
+
it, so the SQL below is what a default install actually provisions. The owner is
|
|
11
|
+
`memhouse_root` — the **database** was shortened to `mem`, the owning role was not.
|
|
12
|
+
|
|
13
|
+
They must agree or rooms get provisioned somewhere the product never reads. Override with
|
|
14
|
+
`MEM_DB` / `MEM_USER` (or the `MEMHOUSE_*` equivalents); `provision.js` prints the
|
|
15
|
+
connection it resolved so a mismatch is visible rather than silent.
|
|
16
|
+
|
|
17
|
+
An install predating this default carries `MEMHOUSE_DB` in `~/.memhouse/env`, and explicit
|
|
18
|
+
config outranks the default, so nothing repoints you at a different house. **That does not
|
|
19
|
+
mean a 0.3.x house keeps working** — the database name survives the upgrade, the tables do
|
|
20
|
+
not. See [INSTALL.md](INSTALL.md#upgrading-from-03x--read-this-before-you-do).
|
|
21
|
+
|
|
22
|
+
## The grant set
|
|
23
|
+
|
|
24
|
+
Two statements per room. The split is the point.
|
|
25
|
+
|
|
26
|
+
```sql
|
|
27
|
+
GRANT ALL ON mem.sessions_<m> TO <m>;
|
|
28
|
+
GRANT SELECT ON mem.sessions_<m> TO <m> WITH GRANT OPTION;
|
|
29
|
+
GRANT ALL ON mem.messages_<m> TO <m>;
|
|
30
|
+
GRANT SELECT ON mem.messages_<m> TO <m> WITH GRANT OPTION;
|
|
31
|
+
GRANT ALL ON mem.tool_calls_<m> TO <m>;
|
|
32
|
+
GRANT SELECT ON mem.tool_calls_<m> TO <m> WITH GRANT OPTION;
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Plus `SELECT` on whichever Merge rooms exist, which is what makes the team room narrow to
|
|
36
|
+
the caller's own rooms instead of denying outright.
|
|
37
|
+
|
|
38
|
+
**`ALL`, not re-grantable.** The member owns their room outright — `DROP` and `TRUNCATE`
|
|
39
|
+
included, because it is their memory to destroy. It also covers the mutation privileges
|
|
40
|
+
below without naming them, which is what stops the version-dependent trap.
|
|
41
|
+
|
|
42
|
+
**`SELECT`, re-grantable.** A share is read-only **by construction**. `GRANT ALL … WITH
|
|
43
|
+
GRANT OPTION` expands to 45 privileges on 26.7 — `DROP TABLE`, `TRUNCATE`,
|
|
44
|
+
`CREATE ROW POLICY`, `SYSTEM DROP REPLICA` among them — every one of which a member could
|
|
45
|
+
then hand to a colleague while meaning "let them read my sessions". Measured on 26.7.2:
|
|
46
|
+
`GRANT SELECT … TO bob` succeeds, `GRANT DROP TABLE … TO bob` is refused 497, and the
|
|
47
|
+
member can still drop their own room.
|
|
48
|
+
|
|
49
|
+
Never `ON mem.*`. A member who can read `mem.*` can read every other member's rooms, and
|
|
50
|
+
then this is a shared house with longer table names.
|
|
51
|
+
|
|
52
|
+
### Why the mutation grants are not optional
|
|
53
|
+
|
|
54
|
+
v4 said `SELECT, INSERT`. A member provisioned that way ships **once** and then fails on
|
|
55
|
+
every subsequent pass.
|
|
56
|
+
|
|
57
|
+
Both `ALTER` privileges are needed, and **which one bites depends on the server version**.
|
|
58
|
+
`DELETE FROM` is a *lightweight* delete, implemented as
|
|
59
|
+
`ALTER TABLE ... UPDATE _row_exists = 0`: ClickHouse 26.7 refused it for want of
|
|
60
|
+
`ALTER DELETE`, and 25.11 refused the same statement for want of
|
|
61
|
+
`ALTER UPDATE(_row_exists)`. Granting the one you happened to test against produces a
|
|
62
|
+
member who cannot ship on the other server, so grant both.
|
|
63
|
+
|
|
64
|
+
The shipper is not append-only. A session file grows while the agent works, so each pass
|
|
65
|
+
re-parses the whole transcript and re-ships it; to keep that idempotent it **clears the
|
|
66
|
+
session's existing rows and re-inserts them**. `ReplacingMergeTree(ingested_at)` collapses
|
|
67
|
+
duplicates by key eventually, but it cannot remove a row the new parse no longer produces —
|
|
68
|
+
a re-parse that yields fewer rows would otherwise leave a stale `seq` tail behind forever.
|
|
69
|
+
So the clear is a real `ALTER TABLE ... DELETE`, and without the privilege the first
|
|
70
|
+
incremental pass dies with
|
|
71
|
+
|
|
72
|
+
```
|
|
73
|
+
Not enough privileges. To execute this query, it's necessary to have the grant
|
|
74
|
+
ALTER UPDATE(_row_exists) ON mem.messages_<m>
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
The failure is invisible in a first-run test, which is exactly why it survived four
|
|
78
|
+
document versions: provision, ship, count rows, declare victory. It only appears on the
|
|
79
|
+
*second* ship of a *changed* session.
|
|
80
|
+
|
|
81
|
+
### The delete predicate must bind the user, not call `currentUser()`
|
|
82
|
+
|
|
83
|
+
Related, and worse, because it fails **silently**. The clear was written as
|
|
84
|
+
|
|
85
|
+
```sql
|
|
86
|
+
DELETE FROM messages_<m> WHERE session_id = {id:String} AND user_id = currentUser()
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
A mutation does not evaluate `currentUser()` in the caller's context. Measured on
|
|
90
|
+
ClickHouse 25.11 and on chdb: the identical predicate with the literal value deleted 2000
|
|
91
|
+
rows where `currentUser()` deleted **0** — no error, no warning, nothing in the log. The
|
|
92
|
+
stale tail the delete exists to remove simply survives. The value is bound instead, read
|
|
93
|
+
from the server over the same connection, so it is the same identity with none of the
|
|
94
|
+
context dependence. This affected the **shared** layout too, and predates this fork.
|
|
95
|
+
|
|
96
|
+
### What the privilege lets a member do
|
|
97
|
+
|
|
98
|
+
Delete and mutate rows in their own rooms. That is their own memory, and they can already
|
|
99
|
+
overwrite it by re-shipping — so this widens what a member can destroy about themselves,
|
|
100
|
+
and nothing about anyone else. Grants remain per-room and explicit, so `ALTER DELETE` on
|
|
101
|
+
`messages_alice` says nothing about `messages_bob`.
|
|
102
|
+
|
|
103
|
+
`DROP TABLE` is included too (see below): a member can remove their own room, not just
|
|
104
|
+
empty it.
|
|
105
|
+
|
|
106
|
+
### Consequence of `WITH GRANT OPTION`
|
|
107
|
+
|
|
108
|
+
Grant-option covers every privilege **in the statement it is attached to**, which is why
|
|
109
|
+
it is attached to the `SELECT` statement and not the `ALL` one. Alice can hand bob read
|
|
110
|
+
access to `messages_alice` and nothing else; she cannot hand him the mutation or drop
|
|
111
|
+
rights even by mistake. The convention `SHARING` describes is now enforced by the engine
|
|
112
|
+
rather than by everyone remembering it.
|
|
113
|
+
|
|
114
|
+
A share is the three rooms and nothing else. The recipient runs the same rollup query over
|
|
115
|
+
them under their own credential, so there is no fourth grant to forget — which is the
|
|
116
|
+
concrete reason the query beat the view.
|
|
117
|
+
|
|
118
|
+
`WITH GRANT OPTION` is what makes whole-room sharing self-serve. Without it every share
|
|
119
|
+
becomes an operator ticket, which is most of the reason this fork exists.
|
|
120
|
+
|
|
121
|
+
The owner (`memhouse_root` — the house was shortened to `mem`, the owning role was not)
|
|
122
|
+
holds `ALL ON mem.* WITH GRANT OPTION`, so a second member joining
|
|
123
|
+
an existing house needs no new owner setup — house, owner, and Merge rooms are already
|
|
124
|
+
there.
|
|
125
|
+
|
|
126
|
+
**RESOLVED:** the member does get `CREATE TABLE` / `DROP TABLE` on their own rooms — it
|
|
127
|
+
is inside `ALL`. That is what lets `--ensure-schema` run as the member and makes the solo
|
|
128
|
+
install one command with no owner involvement. The cost is accepted: a member can drop
|
|
129
|
+
their own memory. The thing that was actually worth preventing — a member handing those
|
|
130
|
+
rights to someone else — is prevented by the grant-option split above.
|
|
131
|
+
|
|
132
|
+
## Kernel path
|
|
133
|
+
|
|
134
|
+
**Not finalized.** The realm holds the privileged credential and mints members; what the
|
|
135
|
+
installer asks it for and what it gets back is not designed yet.
|
|
136
|
+
|
|
137
|
+
What is settled regardless: the room layout, the grant set above, and the identity rule
|
|
138
|
+
(the ClickHouse user is the identity). What is open is only who plays admin and how the
|
|
139
|
+
handle is chosen. Whatever that turns out to be, provisioning wants to be ONE idempotent
|
|
140
|
+
step — three rooms plus their grants — because it is also the hook for rolling a schema
|
|
141
|
+
change across rooms.
|
|
142
|
+
|
|
143
|
+
Nothing touches the Merge rooms on member join: they are a regex over room names and
|
|
144
|
+
**auto-discover** rooms created after them.
|
|
145
|
+
|
|
146
|
+
## Standalone path
|
|
147
|
+
|
|
148
|
+
No kernel, so no capability layer; the CLI performs the same steps. `provision.js` is that
|
|
149
|
+
path today: it creates the house with a client that has *not* selected it (on a fresh
|
|
150
|
+
server the database does not exist yet, and selecting it fails before it can be created),
|
|
151
|
+
then the rooms, then the grants.
|
|
152
|
+
|
|
153
|
+
> **OPEN:** a standalone user is simultaneously owner and member. One credential holding
|
|
154
|
+
> both roles, or two? Today's `memhouse install` assumes a single credential that owns
|
|
155
|
+
> everything, which no longer maps cleanly.
|
|
156
|
+
|
|
157
|
+
## Row-policy rights
|
|
158
|
+
|
|
159
|
+
`GRANT ACCESS MANAGEMENT ON mem.*` expands to
|
|
160
|
+
`CREATE/ALTER/DROP/SHOW ROW POLICY ON mem.*` — house-scoped, no global access management
|
|
161
|
+
required. This is what lets the owner serve partial-share requests.
|
|
162
|
+
|
|
163
|
+
**Owner only.** The scope is the house, not a room, so any holder could attach a policy to
|
|
164
|
+
another member's rooms. Policies filter rather than grant, so it is not a data leak — but
|
|
165
|
+
it is a denial of service on a colleague's own reads.
|
|
166
|
+
|
|
167
|
+
## `EXECUTE AS`
|
|
168
|
+
|
|
169
|
+
Owner may hold it; **members must not**. It changes `currentUser()`, so a member holding
|
|
170
|
+
it could write rows stamped as someone else and defeat the provenance the schema relies
|
|
171
|
+
on.
|
|
172
|
+
|
|
173
|
+
## Not yet written
|
|
174
|
+
|
|
175
|
+
- the capability script
|
|
176
|
+
- member removal: drop the three rooms, revoke, and decide what happens to rooms they had
|
|
177
|
+
shared outward
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# per-member rooms
|
|
2
|
+
|
|
3
|
+
**One house, a set of rooms per member.** This replaced a shared layout — three rooms
|
|
4
|
+
named `sessions`/`messages`/`tool_calls` for everyone, separated by row policies — which
|
|
5
|
+
has been removed. There is no switch and no fallback: this is the data model.
|
|
6
|
+
|
|
7
|
+
**The design is implemented** — `rooms.js`, `provision.js`, the two DDL templates, and
|
|
8
|
+
room routing through the read layer — and proven end to end on ClickHouse 26.7.1 and
|
|
9
|
+
25.11.
|
|
10
|
+
|
|
11
|
+
| Document | What |
|
|
12
|
+
|---|---|
|
|
13
|
+
| [`PLAN.md`](PLAN.md) | the model, why, phases, measured facts |
|
|
14
|
+
| [`PROVISIONING.md`](PROVISIONING.md) | the grant set (three per member), kernel capability, standalone path |
|
|
15
|
+
| [`SCHEMA.md`](SCHEMA.md) | room naming, shapes, sort keys, Merge rooms, and why `sessions_v` is a saved query |
|
|
16
|
+
| [`SHARING.md`](SHARING.md) | whole-room self-serve (three grants), partial rows via the owner |
|
|
17
|
+
|
|
18
|
+
These four documents describe the design as it stands. Earlier revisions are in
|
|
19
|
+
`git log`, not in this directory — a superseded document kept beside the current one is a
|
|
20
|
+
contradiction waiting to be quoted, and this directory produced two review findings that
|
|
21
|
+
way before the versions were removed.
|
|
22
|
+
|
|
23
|
+
## The one-paragraph version
|
|
24
|
+
|
|
25
|
+
The house is **`mem`**. Each member gets `mem.sessions_<member>`, `mem.messages_<member>`,
|
|
26
|
+
`mem.tool_calls_<member>`, with three grants over them —
|
|
27
|
+
`SELECT, INSERT, ALTER UPDATE, ALTER DELETE`, all `WITH GRANT OPTION`. The session rollup
|
|
28
|
+
is a saved query over those same rooms, not a fourth object. Isolation stops
|
|
29
|
+
being a row policy that must be right everywhere and becomes a grant that is simply
|
|
30
|
+
absent — it fails closed. Sharing a whole room needs no operator. Team-wide reads come
|
|
31
|
+
from `Merge` rooms anchored on the room type (`^messages_`), which reduce to each caller's
|
|
32
|
+
own grants, auto-discover new members, and tolerate schema drift — provided the caller
|
|
33
|
+
holds `SELECT` on the Merge room itself, which provisioning issues; without it the team
|
|
34
|
+
room denies instead of narrowing.
|
|
35
|
+
`user_id MATERIALIZED currentUser()` stays on every room: unforgeable by members, who are
|
|
36
|
+
not granted `EXECUTE AS`.
|
|
37
|
+
|
|
38
|
+
## Biggest open questions
|
|
39
|
+
|
|
40
|
+
1. **May members create and drop their own rooms?** It works and is safely scoped, but it
|
|
41
|
+
lets a member drop their own memory and makes schema rollout non-central.
|
|
42
|
+
2. **Key order** — is `user_id` still worth an `ORDER BY` slot now the room is the tenant?
|
|
43
|
+
3. **Standalone path** — a user with no kernel is both owner and member; one credential or
|
|
44
|
+
two?
|
|
45
|
+
4. **Is `EXECUTE AS` scopeable** to `mem.*` rather than `*.*`?
|
|
46
|
+
5. **Is a saved rollup enough for agents?** It has no name to type. `memhouse
|
|
47
|
+
sessions-query` prints it, which is a worse affordance than a name and the price of
|
|
48
|
+
having nothing to own, grant or collide.
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
# SCHEMA — rooms, keys, Merge rooms, the rollup
|
|
2
|
+
The room shapes, keys and Merge rooms are
|
|
3
|
+
unchanged. What changes is that v4's decision about `sessions_v` — *a saved query, not a
|
|
4
|
+
room* — is now the implemented one, after a detour through a stored view that this
|
|
5
|
+
version records rather than hides.
|
|
6
|
+
|
|
7
|
+
## `sessions_v` is a saved query, not a room
|
|
8
|
+
|
|
9
|
+
v4 said so. Round 1 of the review campaign implemented it as a **stored view** instead —
|
|
10
|
+
reactively, to fix a finding that the read layer named the shared rooms — without going
|
|
11
|
+
back to this file. Everything that followed was the cost of that override:
|
|
12
|
+
|
|
13
|
+
- the view needed a name, and `sessions_v_<m>` sits inside the `^sessions_` pattern the
|
|
14
|
+
Merge rooms select on. `all_sessions` silently absorbed every member's aggregate view
|
|
15
|
+
alongside their base room: **161 sessions counted as 322**.
|
|
16
|
+
- so the name moved to `v_sessions_<m>`, which then needed a reserved prefix so no member
|
|
17
|
+
handle could re-create the collision;
|
|
18
|
+
- and a fourth grant per member, because a recipient given the three rooms could read rows
|
|
19
|
+
and use none of the product's read paths;
|
|
20
|
+
- and a fourth object to create, roll forward, check in `doctor`, and eventually drop.
|
|
21
|
+
|
|
22
|
+
None of that buys anything the query does not already do. Reverted.
|
|
23
|
+
|
|
24
|
+
### What it is now
|
|
25
|
+
|
|
26
|
+
`rooms.js` exposes `sessions_v` as **SQL text** — a parenthesised `SELECT` over the
|
|
27
|
+
caller's own rooms. It drops into the same `FROM … AS c` position a view name would have
|
|
28
|
+
occupied, so the read layer treats it as just another resolved room name and the
|
|
29
|
+
substitution point is one function. (The shared layout, where this key resolved to a
|
|
30
|
+
stored view's *name*, is gone; the text form is the only one now.)
|
|
31
|
+
|
|
32
|
+
```sql
|
|
33
|
+
FROM {{sessions_v}} AS c -- resolves to: FROM ( SELECT … FROM sessions_alice … ) AS c
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
It runs **under the caller's credential**, substituted with the caller's own room names,
|
|
37
|
+
so it inherits their grants exactly and raises no view-ownership question at all.
|
|
38
|
+
|
|
39
|
+
### The two properties that had to survive
|
|
40
|
+
|
|
41
|
+
- the join is a **LEFT JOIN**, so a session with no messages still appears with zero
|
|
42
|
+
aggregates instead of vanishing from every count;
|
|
43
|
+
- `join_use_nulls = 1` makes the unmatched message columns NULL, so `coalesce` yields true
|
|
44
|
+
zeros rather than counting the placeholder row.
|
|
45
|
+
|
|
46
|
+
The view carried the second as a trailing `SETTINGS` clause. **A subquery cannot**, so it
|
|
47
|
+
moved to the caller: `READ_SETTINGS = { final: 1, join_use_nulls: 1 }`, applied by the
|
|
48
|
+
dashboard client, the shipper and the CLI's raw HTTP alike. A unit check asserts the text
|
|
49
|
+
carries no `SETTINGS` of its own, because putting it back would break every read that uses
|
|
50
|
+
the rollup.
|
|
51
|
+
|
|
52
|
+
### The cost, stated plainly
|
|
53
|
+
|
|
54
|
+
An agent writing ad-hoc SQL has no name to type. `memhouse sessions-query` prints the
|
|
55
|
+
rollup resolved for whoever the configured credential is, and the three delivery skills
|
|
56
|
+
point at it. That is a worse affordance than a name, and it is the price of the property
|
|
57
|
+
that made v4 choose a query: nothing to own, nothing to grant, nothing in a namespace.
|
|
58
|
+
|
|
59
|
+
## Room naming, keys, Merge rooms
|
|
60
|
+
|
|
61
|
+
Unchanged from v4. Rooms are type-first — `sessions_<m>`, `messages_<m>`,
|
|
62
|
+
`tool_calls_<m>` — so the Merge rooms anchor on a fixed room type and never match
|
|
63
|
+
themselves. With the view gone, `^sessions_` now matches **only** base session rooms,
|
|
64
|
+
which is what it was always supposed to mean.
|
|
65
|
+
|
|
66
|
+
> **OPEN** (carried from v4): is `all_sessions` a Merge over the member `sessions_*`
|
|
67
|
+
> rooms, or a saved query over `all_messages`? The first is cheaper; the second guarantees
|
|
68
|
+
> the aggregates match the per-member rollup exactly.
|
|
69
|
+
|
|
70
|
+
## Not yet written
|
|
71
|
+
|
|
72
|
+
- the kernel capability that provisions a member
|
|
73
|
+
- member removal: drop the three rooms, revoke, decide what happens to rooms shared outward
|
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
# SHARING — two tiers
|
|
2
|
+
|
|
3
|
+
**A share is three grants**, one per room, and there is no fourth: the rollup is a saved
|
|
4
|
+
query over exactly those rooms rather than a stored object, so a recipient who can read
|
|
5
|
+
the rooms can run it.
|
|
6
|
+
|
|
7
|
+
A share is also **read-only by construction**. `WITH GRANT OPTION` is attached to the
|
|
8
|
+
member's `SELECT` grant and not to their `ALL` grant ([PROVISIONING](PROVISIONING.md)),
|
|
9
|
+
so alice cannot hand bob mutation or drop rights on her room even by mistake.
|
|
10
|
+
|
|
11
|
+
## Tier 1 — whole room, self-serve
|
|
12
|
+
|
|
13
|
+
A member holds `WITH GRANT OPTION` on their own three rooms — there is no fourth object
|
|
14
|
+
to hold anything on — so they share without an operator:
|
|
15
|
+
|
|
16
|
+
```sql
|
|
17
|
+
GRANT SELECT ON mem.messages_alice TO bob;
|
|
18
|
+
GRANT SELECT ON mem.sessions_alice TO bob;
|
|
19
|
+
GRANT SELECT ON mem.tool_calls_alice TO bob;
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
**Three statements.** The rollup every read path uses is a saved query over exactly these
|
|
23
|
+
rooms, run under the recipient's own credential — so a recipient who can read the rooms
|
|
24
|
+
can run it, and there is no fourth grant to forget. That is the concrete payoff of the
|
|
25
|
+
query over the view, and it is why SCHEMA v4 chose it before the implementation briefly
|
|
26
|
+
went the other way.
|
|
27
|
+
|
|
28
|
+
`memhouse sessions-query` prints the rollup for whoever is connected, which is what a
|
|
29
|
+
recipient pastes into a `FROM (...) AS c` when writing ad-hoc SQL.
|
|
30
|
+
|
|
31
|
+
This is the tier per-member rooms exist to enable. Under a row policy the same act needs
|
|
32
|
+
`CREATE ROW POLICY`, which members do not have and the kernel withholds from agency
|
|
33
|
+
owners by default.
|
|
34
|
+
|
|
35
|
+
Bob reading `messages_alice` sees `user_id = 'alice'` on every row. That stamp is
|
|
36
|
+
unforgeable **by members**, since `EXECUTE AS` is a grantable privilege they do not hold,
|
|
37
|
+
and forgeable by the owner or anyone granted it. Trust boundary is the owner, not the
|
|
38
|
+
engine.
|
|
39
|
+
|
|
40
|
+
Bob's Merge-room reads widen automatically once alice grants him a room — `Merge` reduces
|
|
41
|
+
to the caller's actual grants, so nothing else needs changing. Measured end to end on
|
|
42
|
+
25.11: alice's `GRANT` took what bob sees of her rows through `all_messages` from 0 to
|
|
43
|
+
22,413, and her
|
|
44
|
+
`REVOKE` returned it to 0, and his direct read was denied again.
|
|
45
|
+
|
|
46
|
+
**A member must hold SELECT on the Merge rooms themselves** for any of that to work —
|
|
47
|
+
provisioning issues it. Without that grant the team room does not narrow to what the
|
|
48
|
+
caller can read, it denies outright, which looks identical to isolation and is not.
|
|
49
|
+
|
|
50
|
+
### Grant-option covers the whole grant
|
|
51
|
+
|
|
52
|
+
Alice's grant-option is over `SELECT, INSERT, ALTER UPDATE, ALTER DELETE`, so she *can*
|
|
53
|
+
hand bob the mutation rights on her own rooms, not merely `SELECT`. Her data, her call —
|
|
54
|
+
but a share is `SELECT`-only by convention, and nothing in the engine enforces the
|
|
55
|
+
convention. A `share` verb should encode it.
|
|
56
|
+
|
|
57
|
+
> **OPEN:** a `share` / `unshare` CLI verb wrapping the three grants, so a member never
|
|
58
|
+
> types SQL. Mirrors memory-house's `mh-share`.
|
|
59
|
+
|
|
60
|
+
## Tier 2 — some rows, owner-mediated
|
|
61
|
+
|
|
62
|
+
Row policies remain the only way to share a *subset*. The member asks the owner, who
|
|
63
|
+
holds house-scoped policy rights:
|
|
64
|
+
|
|
65
|
+
```sql
|
|
66
|
+
GRANT SELECT ON mem.messages_alice TO bob; -- privilege
|
|
67
|
+
CREATE ROW POLICY share_alice_to_bob ON mem.messages_alice
|
|
68
|
+
FOR SELECT USING <predicate> TO bob; -- restriction
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Both halves are required: a policy filters, it never grants. Privilege without a policy
|
|
72
|
+
shares everything; a policy without privilege shares nothing.
|
|
73
|
+
|
|
74
|
+
> **OPEN:** which predicates — by project, by date range, by `source`? Free-form SQL from
|
|
75
|
+
> a member is an injection surface, so this likely wants a small fixed vocabulary.
|
|
76
|
+
|
|
77
|
+
> **OPEN:** what a partial recipient's rollup should show. The rollup is a saved query
|
|
78
|
+
> over `sessions_alice` and `messages_alice`, so a policy on `messages_alice` filters the
|
|
79
|
+
> rows it aggregates and the recipient sees totals computed over their own slice — which
|
|
80
|
+
> is arguably right, and is untested. Nothing extra has to be shared for it to work,
|
|
81
|
+
> which is one fewer moving part than the stored view had.
|
|
82
|
+
|
|
83
|
+
## Why the tiers are asymmetric
|
|
84
|
+
|
|
85
|
+
Deliberate. Whole-room sharing is the common case, is coarse, and is safe to self-serve —
|
|
86
|
+
the worst outcome is oversharing your own data. Partial sharing writes a filter into the
|
|
87
|
+
house's access model, so it goes through the owner.
|
|
88
|
+
|
|
89
|
+
## What sharing does not do
|
|
90
|
+
|
|
91
|
+
**It is not a boundary between agents sharing one credential.** A crew running under a
|
|
92
|
+
single member credential reads and writes the same rooms; any per-agent tag is
|
|
93
|
+
client-supplied. Enforced separation needs separate ClickHouse users, which means
|
|
94
|
+
separate rooms — this design one level down. Deferred with `agent_id`.
|
|
95
|
+
|
|
96
|
+
## Not yet written
|
|
97
|
+
|
|
98
|
+
- `memhouse share <member>` / `unshare`
|
|
99
|
+
- how the dashboard and skills list rooms a caller can read but does not own — filter
|
|
100
|
+
`system.tables` by grants (it *is* grant-filtered: alice sees exactly her own three
|
|
101
|
+
three rooms, measured), or read the Merge rooms and group by `_table`
|
|
102
|
+
- revocation when a member is removed while their rooms are shared outward
|