memhouse 0.7.1 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +79 -43
- package/bin/memhouse.js +153 -147
- package/editors/antigravity.js +7 -15
- package/editors/cursor.js +18 -24
- package/editors/goose.js +9 -21
- package/editors/index.js +19 -70
- package/editors/opencode.js +6 -15
- package/editors/sqlite.js +74 -0
- package/editors/windsurf.js +3 -3
- package/editors/zed.js +10 -15
- package/memhouse/COMPETITION.md +9 -5
- package/memhouse/DESIGN.md +29 -30
- package/memhouse/RESUME-AND-UPDATE.md +7 -10
- package/memhouse/delivery/AGENT-INSTALL.md +15 -13
- package/memhouse/delivery/PROMPT.md +7 -9
- package/memhouse/delivery/kernel-install.md +18 -33
- package/memhouse/delivery/plugin/skills/search/SKILL.md +25 -38
- package/memhouse/delivery/plugin/skills/sessions/SKILL.md +15 -28
- package/memhouse/delivery/plugin/skills/sql/SKILL.md +23 -36
- package/memhouse/host.js +1 -1
- package/memhouse/house/HOUSE.md +74 -0
- package/memhouse/house/house.js +170 -0
- package/memhouse/{per-member/schema-member.sql.tpl → house/schema.sql.tpl} +8 -11
- package/memhouse/resume.js +1 -1
- package/memhouse/self-update.js +4 -4
- package/memhouse/server/queries.js +8 -2
- package/memhouse/shipper/ship.js +30 -37
- package/package.json +2 -6
- package/memhouse/per-member/INSTALL.md +0 -228
- package/memhouse/per-member/PLAN.md +0 -174
- package/memhouse/per-member/PROVISIONING.md +0 -177
- package/memhouse/per-member/README.md +0 -48
- package/memhouse/per-member/SCHEMA.md +0 -120
- package/memhouse/per-member/SHARING.md +0 -149
- package/memhouse/per-member/provision.js +0 -297
- package/memhouse/per-member/rooms.js +0 -252
- package/memhouse/per-member/schema-merge.sql.tpl +0 -24
|
@@ -1,120 +0,0 @@
|
|
|
1
|
-
# SCHEMA — rooms, keys, Merge rooms, the rollup
|
|
2
|
-
The room shapes, keys and Merge rooms are
|
|
3
|
-
unchanged. What changes is that v4's decision about `sessions_v` — *a saved query, not a
|
|
4
|
-
room* — is now the implemented one, after a detour through a stored view that this
|
|
5
|
-
version records rather than hides.
|
|
6
|
-
|
|
7
|
-
## `sessions_v` is a saved query, not a room
|
|
8
|
-
|
|
9
|
-
v4 said so. Round 1 of the review campaign implemented it as a **stored view** instead —
|
|
10
|
-
reactively, to fix a finding that the read layer named the shared rooms — without going
|
|
11
|
-
back to this file. Everything that followed was the cost of that override:
|
|
12
|
-
|
|
13
|
-
- the view needed a name, and `sessions_v_<m>` sits inside the `^sessions_` pattern the
|
|
14
|
-
Merge rooms select on. `all_sessions` silently absorbed every member's aggregate view
|
|
15
|
-
alongside their base room: **161 sessions counted as 322**.
|
|
16
|
-
- so the name moved to `v_sessions_<m>`, which then needed a reserved prefix so no member
|
|
17
|
-
handle could re-create the collision;
|
|
18
|
-
- and a fourth grant per member, because a recipient given the three rooms could read rows
|
|
19
|
-
and use none of the product's read paths;
|
|
20
|
-
- and a fourth object to create, roll forward, check in `doctor`, and eventually drop.
|
|
21
|
-
|
|
22
|
-
None of that buys anything the query does not already do. Reverted.
|
|
23
|
-
|
|
24
|
-
### What it is now
|
|
25
|
-
|
|
26
|
-
`rooms.js` exposes `sessions_v` as **SQL text** — a parenthesised `SELECT` over the
|
|
27
|
-
caller's own rooms. It drops into the same `FROM … AS c` position a view name would have
|
|
28
|
-
occupied, so the read layer treats it as just another resolved room name and the
|
|
29
|
-
substitution point is one function. (The shared layout, where this key resolved to a
|
|
30
|
-
stored view's *name*, is gone; the text form is the only one now.)
|
|
31
|
-
|
|
32
|
-
```sql
|
|
33
|
-
FROM {{sessions_v}} AS c -- resolves to: FROM ( SELECT … FROM sessions_alice … ) AS c
|
|
34
|
-
```
|
|
35
|
-
|
|
36
|
-
It runs **under the caller's credential**, substituted with the caller's own room names,
|
|
37
|
-
so it inherits their grants exactly and raises no view-ownership question at all.
|
|
38
|
-
|
|
39
|
-
### The two properties that had to survive
|
|
40
|
-
|
|
41
|
-
- the join is a **LEFT JOIN**, so a session with no messages still appears with zero
|
|
42
|
-
aggregates instead of vanishing from every count;
|
|
43
|
-
- `join_use_nulls = 1` makes the unmatched message columns NULL, so `coalesce` yields true
|
|
44
|
-
zeros rather than counting the placeholder row.
|
|
45
|
-
|
|
46
|
-
The view carried the second as a trailing `SETTINGS` clause. **A subquery cannot**, so it
|
|
47
|
-
moved to the caller: `READ_SETTINGS = { final: 1, join_use_nulls: 1 }`, applied by the
|
|
48
|
-
dashboard client, the shipper and the CLI's raw HTTP alike. A unit check asserts the text
|
|
49
|
-
carries no `SETTINGS` of its own, because putting it back would break every read that uses
|
|
50
|
-
the rollup.
|
|
51
|
-
|
|
52
|
-
### The cost, stated plainly
|
|
53
|
-
|
|
54
|
-
An agent writing ad-hoc SQL has no name to type. `memhouse sessions-query` prints the
|
|
55
|
-
rollup resolved for whoever the configured credential is, and the three delivery skills
|
|
56
|
-
point at it. That is a worse affordance than a name, and it is the price of the property
|
|
57
|
-
that made v4 choose a query: nothing to own, nothing to grant, nothing in a namespace.
|
|
58
|
-
|
|
59
|
-
## Room naming, keys, Merge rooms
|
|
60
|
-
|
|
61
|
-
Unchanged from v4. Rooms are type-first — `sessions_<m>`, `messages_<m>`,
|
|
62
|
-
`tool_calls_<m>` — so the Merge rooms anchor on a fixed room type and never match
|
|
63
|
-
themselves. With the view gone, `^sessions_` now matches **only** base session rooms,
|
|
64
|
-
which is what it was always supposed to mean.
|
|
65
|
-
|
|
66
|
-
> **OPEN** (carried from v4): is `all_sessions` a Merge over the member `sessions_*`
|
|
67
|
-
> rooms, or a saved query over `all_messages`? The first is cheaper; the second guarantees
|
|
68
|
-
> the aggregates match the per-member rollup exactly.
|
|
69
|
-
|
|
70
|
-
## Not yet written
|
|
71
|
-
|
|
72
|
-
- the kernel capability that provisions a member
|
|
73
|
-
- member removal: drop the three rooms, revoke, decide what happens to rooms shared outward
|
|
74
|
-
|
|
75
|
-
## What a member can still learn about the others
|
|
76
|
-
|
|
77
|
-
Isolation is a grant that is absent, so no member can read another's rows — verified by
|
|
78
|
-
trying every read and write against another member's rooms, plus `system.parts`,
|
|
79
|
-
`system.query_log`, `system.processes` and the rest, all of which refuse. Two things do
|
|
80
|
-
leak, both structural rather than bugs in the code, and neither is fixed by tightening a
|
|
81
|
-
grant:
|
|
82
|
-
|
|
83
|
-
**The roster.** A member's handle is part of their table name, and ClickHouse distinguishes
|
|
84
|
-
"forbidden" from "absent": `SELECT count() FROM messages_bob` returns `Code: 497
|
|
85
|
-
ACCESS_DENIED` (HTTP 500) if bob exists and `Code: 60 UNKNOWN_TABLE` (HTTP 404) if he does
|
|
86
|
-
not. So any member can enumerate the membership of the house by guessing handles.
|
|
87
|
-
`system.tables` correctly hides the other rooms; this reopens it. `EXISTS TABLE` does not
|
|
88
|
-
leak — it answers 497 either way — so it is specifically the natural probe that talks. If
|
|
89
|
-
the membership of a house is itself sensitive, the per-member layout is the wrong shape for
|
|
90
|
-
it.
|
|
91
|
-
|
|
92
|
-
**Volume.** `system.tables.total_rows` and `total_bytes` are readable for the Merge rooms,
|
|
93
|
-
and a Merge room spans every member:
|
|
94
|
-
|
|
95
|
-
name total_rows total_bytes
|
|
96
|
-
all_messages 87 42455 <- the whole house
|
|
97
|
-
messages_alice 33 25070 <- what alice can actually read
|
|
98
|
-
|
|
99
|
-
So a member can see the house's combined row count and byte volume, and by polling it, when
|
|
100
|
-
other members are shipping. Withholding `SELECT` on the Merge rooms is not the fix — it
|
|
101
|
-
makes the team room deny rather than narrow, which is the property those rooms exist for.
|
|
102
|
-
|
|
103
|
-
## Why members are not granted ALL on their own rooms
|
|
104
|
-
|
|
105
|
-
`GRANT ALL ON <db>.messages_<m>` expands to 45 privileges on 26.7, and one of them is
|
|
106
|
-
`CREATE TABLE` **on that name**. A member would then own the name rather than the data:
|
|
107
|
-
|
|
108
|
-
DROP TABLE messages_alice;
|
|
109
|
-
CREATE TABLE messages_alice AS all_messages ENGINE = Merge('<db>','^messages_');
|
|
110
|
-
|
|
111
|
-
`all_messages` is `Merge(db,'^messages_')`, so it now contains a Merge over its own
|
|
112
|
-
namespace and every other member's rows are counted twice in the team room — measured, a
|
|
113
|
-
house reading alice 6030 / bob 2401 / carol 12 became bob 4802 / carol 24, with no error.
|
|
114
|
-
Confidentiality survives (the Merge still narrows by grants) but integrity does not, and a
|
|
115
|
-
team-wide number is what the room is for. It is the same failure recorded above under the
|
|
116
|
-
`sessions_v` rename, reached on purpose instead of by accident.
|
|
117
|
-
|
|
118
|
-
Members therefore get exactly what the shipper uses — `SELECT, INSERT, ALTER UPDATE, ALTER
|
|
119
|
-
DELETE, ALTER ADD COLUMN, OPTIMIZE` — and nothing that defines an object. Replacing a room
|
|
120
|
-
is the admin's job.
|
|
@@ -1,149 +0,0 @@
|
|
|
1
|
-
# SHARING — two tiers
|
|
2
|
-
|
|
3
|
-
**A share is three grants**, one per room, and there is no fourth: the rollup is a saved
|
|
4
|
-
query over exactly those rooms rather than a stored object, so a recipient who can read
|
|
5
|
-
the rooms can run it.
|
|
6
|
-
|
|
7
|
-
A share is also **read-only by construction**. `WITH GRANT OPTION` is attached to the
|
|
8
|
-
member's `SELECT` grant and not to their `ALL` grant ([PROVISIONING](PROVISIONING.md)),
|
|
9
|
-
so alice cannot hand bob mutation or drop rights on her room even by mistake.
|
|
10
|
-
|
|
11
|
-
## Tier 1 — whole room, self-serve
|
|
12
|
-
|
|
13
|
-
A member holds `WITH GRANT OPTION` on their own three rooms — there is no fourth object
|
|
14
|
-
to hold anything on — so they share without an operator:
|
|
15
|
-
|
|
16
|
-
```sql
|
|
17
|
-
GRANT SELECT ON mem.messages_alice TO bob;
|
|
18
|
-
GRANT SELECT ON mem.sessions_alice TO bob;
|
|
19
|
-
GRANT SELECT ON mem.tool_calls_alice TO bob;
|
|
20
|
-
```
|
|
21
|
-
|
|
22
|
-
**Three statements.** The rollup every read path uses is a saved query over exactly these
|
|
23
|
-
rooms, run under the recipient's own credential — so a recipient who can read the rooms
|
|
24
|
-
can run it, and there is no fourth grant to forget. That is the concrete payoff of the
|
|
25
|
-
query over the view, and it is why SCHEMA v4 chose it before the implementation briefly
|
|
26
|
-
went the other way.
|
|
27
|
-
|
|
28
|
-
`memhouse sessions-query` prints the rollup for whoever is connected, which is what a
|
|
29
|
-
recipient pastes into a `FROM (...) AS c` when writing ad-hoc SQL.
|
|
30
|
-
|
|
31
|
-
This is the tier per-member rooms exist to enable. Under a row policy the same act needs
|
|
32
|
-
`CREATE ROW POLICY`, which members do not have and the kernel withholds from agency
|
|
33
|
-
owners by default.
|
|
34
|
-
|
|
35
|
-
Bob reading `messages_alice` sees `user_id = 'alice'` on every row. That stamp is
|
|
36
|
-
unforgeable **by members**, since `EXECUTE AS` is a grantable privilege they do not hold,
|
|
37
|
-
and forgeable by the owner or anyone granted it. Trust boundary is the owner, not the
|
|
38
|
-
engine.
|
|
39
|
-
|
|
40
|
-
The stamp is also pinned **server-side**: the member settings profile carries
|
|
41
|
-
`async_insert = 0 CONST`. A MATERIALIZED column is computed during the insert, and an
|
|
42
|
-
async insert flushes outside that context — measured on 25.11.9.34, `async_insert=1`
|
|
43
|
-
stores `user_id` as the empty string while a sync insert on the same table stamps
|
|
44
|
-
correctly. The shipper always passed `async_insert=0`, so memhouse was never the risk;
|
|
45
|
-
anything else holding a member credential was. With the constraint in force a client that
|
|
46
|
-
*asks* for `async_insert=1` is refused (`SETTING_CONSTRAINT_VIOLATION`), and one that
|
|
47
|
-
simply does not mention the setting is held at 0 — so the insert still succeeds, stamped,
|
|
48
|
-
rather than landing unowned. It is the override that is refused, not the write.
|
|
49
|
-
|
|
50
|
-
Two limits worth stating plainly. The constraint binds the **ClickHouse user**, not the
|
|
51
|
-
database — if you reuse a member's credential for unrelated high-throughput ingestion
|
|
52
|
-
elsewhere on that server, async inserts are refused there too. And it only takes effect
|
|
53
|
-
once the profile is applied: a house provisioned by an older memhouse keeps the old
|
|
54
|
-
profile until any member is provisioned again, which now updates the shared profile for
|
|
55
|
-
everyone at once. `doctor`'s blank-`user_id` check remains as the detector for rows
|
|
56
|
-
written before that.
|
|
57
|
-
|
|
58
|
-
## What the boundary does NOT hide: the roster
|
|
59
|
-
|
|
60
|
-
**Members can discover which other members exist, and this is not fixable here.** A
|
|
61
|
-
handle is a table name, and ClickHouse answers differently for a table you may not read
|
|
62
|
-
than for one that is not there:
|
|
63
|
-
|
|
64
|
-
```
|
|
65
|
-
SELECT count() FROM mem.messages_alice -- exists, not granted → Code 497, Not enough privileges
|
|
66
|
-
SELECT count() FROM mem.messages_nobody -- does not exist → Code 60, Unknown table
|
|
67
|
-
```
|
|
68
|
-
|
|
69
|
-
So probing names enumerates the membership. Measured on 26.7.3.19 that this is
|
|
70
|
-
**ClickHouse's behaviour and not a consequence of our grant set**: a user holding *zero*
|
|
71
|
-
grants anywhere in the database gets exactly the same 497/60 split. No grant, revoke or
|
|
72
|
-
role arrangement changes it.
|
|
73
|
-
|
|
74
|
-
It *is* closable, and the price is the reason it stays open. Suffixing rooms with a random
|
|
75
|
-
token (`messages_alice_8f9b2a`) would still match the `^messages_` Merge selectors, and
|
|
76
|
-
would still leave the share in this document working — alice grants on the room she owns
|
|
77
|
-
and knows, and bob never needs its name. What it costs is that a room name stops being
|
|
78
|
-
**derivable**: `roomNames(member)` is pure today, and every client — the shipper, the
|
|
79
|
-
skills, `sessions-query`, a person typing SQL — would instead have to look its own name up
|
|
80
|
-
first. A real trade, deliberately not taken; not an impossibility.
|
|
81
|
-
|
|
82
|
-
What leaks is **membership, not content**: names and the fact of existence, never rows,
|
|
83
|
-
counts or sizes. In the setting this is built for — a team who already know they are a
|
|
84
|
-
team — that is not a secret. Say so out loud rather than implying the boundary is
|
|
85
|
-
tighter than it is.
|
|
86
|
-
|
|
87
|
-
Bob's Merge-room reads widen automatically once alice grants him a room — `Merge` reduces
|
|
88
|
-
to the caller's actual grants, so nothing else needs changing. Measured end to end on
|
|
89
|
-
25.11: alice's `GRANT` took what bob sees of her rows through `all_messages` from 0 to
|
|
90
|
-
22,413, and her
|
|
91
|
-
`REVOKE` returned it to 0, and his direct read was denied again.
|
|
92
|
-
|
|
93
|
-
**A member must hold SELECT on the Merge rooms themselves** for any of that to work —
|
|
94
|
-
provisioning issues it. Without that grant the team room does not narrow to what the
|
|
95
|
-
caller can read, it denies outright, which looks identical to isolation and is not.
|
|
96
|
-
|
|
97
|
-
### Grant-option covers the whole grant
|
|
98
|
-
|
|
99
|
-
Alice's grant-option is over `SELECT, INSERT, ALTER UPDATE, ALTER DELETE`, so she *can*
|
|
100
|
-
hand bob the mutation rights on her own rooms, not merely `SELECT`. Her data, her call —
|
|
101
|
-
but a share is `SELECT`-only by convention, and nothing in the engine enforces the
|
|
102
|
-
convention. A `share` verb should encode it.
|
|
103
|
-
|
|
104
|
-
> **OPEN:** a `share` / `unshare` CLI verb wrapping the three grants, so a member never
|
|
105
|
-
> types SQL. Mirrors memory-house's `mh-share`.
|
|
106
|
-
|
|
107
|
-
## Tier 2 — some rows, owner-mediated
|
|
108
|
-
|
|
109
|
-
Row policies remain the only way to share a *subset*. The member asks the owner, who
|
|
110
|
-
holds house-scoped policy rights:
|
|
111
|
-
|
|
112
|
-
```sql
|
|
113
|
-
GRANT SELECT ON mem.messages_alice TO bob; -- privilege
|
|
114
|
-
CREATE ROW POLICY share_alice_to_bob ON mem.messages_alice
|
|
115
|
-
FOR SELECT USING <predicate> TO bob; -- restriction
|
|
116
|
-
```
|
|
117
|
-
|
|
118
|
-
Both halves are required: a policy filters, it never grants. Privilege without a policy
|
|
119
|
-
shares everything; a policy without privilege shares nothing.
|
|
120
|
-
|
|
121
|
-
> **OPEN:** which predicates — by project, by date range, by `source`? Free-form SQL from
|
|
122
|
-
> a member is an injection surface, so this likely wants a small fixed vocabulary.
|
|
123
|
-
|
|
124
|
-
> **OPEN:** what a partial recipient's rollup should show. The rollup is a saved query
|
|
125
|
-
> over `sessions_alice` and `messages_alice`, so a policy on `messages_alice` filters the
|
|
126
|
-
> rows it aggregates and the recipient sees totals computed over their own slice — which
|
|
127
|
-
> is arguably right, and is untested. Nothing extra has to be shared for it to work,
|
|
128
|
-
> which is one fewer moving part than the stored view had.
|
|
129
|
-
|
|
130
|
-
## Why the tiers are asymmetric
|
|
131
|
-
|
|
132
|
-
Deliberate. Whole-room sharing is the common case, is coarse, and is safe to self-serve —
|
|
133
|
-
the worst outcome is oversharing your own data. Partial sharing writes a filter into the
|
|
134
|
-
house's access model, so it goes through the owner.
|
|
135
|
-
|
|
136
|
-
## What sharing does not do
|
|
137
|
-
|
|
138
|
-
**It is not a boundary between agents sharing one credential.** A crew running under a
|
|
139
|
-
single member credential reads and writes the same rooms; any per-agent tag is
|
|
140
|
-
client-supplied. Enforced separation needs separate ClickHouse users, which means
|
|
141
|
-
separate rooms — this design one level down. Deferred with `agent_id`.
|
|
142
|
-
|
|
143
|
-
## Not yet written
|
|
144
|
-
|
|
145
|
-
- `memhouse share <member>` / `unshare`
|
|
146
|
-
- how the dashboard and skills list rooms a caller can read but does not own — filter
|
|
147
|
-
`system.tables` by grants (it *is* grant-filtered: alice sees exactly her own three
|
|
148
|
-
three rooms, measured), or read the Merge rooms and group by `_table`
|
|
149
|
-
- revocation when a member is removed while their rooms are shared outward
|
|
@@ -1,297 +0,0 @@
|
|
|
1
|
-
#!/usr/bin/env node
|
|
2
|
-
// mem — provision one member's rooms and grants. Idempotent; also the schema-rollout hook.
|
|
3
|
-
//
|
|
4
|
-
// provision.js --member <name> [--merge] [--owner-grants]
|
|
5
|
-
//
|
|
6
|
-
// Run as the OWNER. Steps, all IF NOT EXISTS / re-runnable:
|
|
7
|
-
// 1. create the member's three rooms from schema-member.sql.tpl
|
|
8
|
-
// 2. grant the member their own three rooms, TWO statements each: MEMBER_PRIVS (not
|
|
9
|
-
// re-grantable), then `SELECT WITH GRANT OPTION`. The split is the point —
|
|
10
|
-
// MEMBER_PRIVS is exactly what the shipper uses and nothing that DEFINES an object,
|
|
11
|
-
// and attaching grant-option to `SELECT` alone makes a share read-only by
|
|
12
|
-
// construction. It is deliberately NOT `ALL`: that includes CREATE TABLE on the
|
|
13
|
-
// member's own room name, which lets them replace the room with a Merge over every
|
|
14
|
-
// member's and double the others' rows in the team room. See rooms.js.
|
|
15
|
-
//
|
|
16
|
-
// THREE ROOMS, not four objects: the session rollup is a saved query over these
|
|
17
|
-
// same rooms, not a stored view, so it needs no object and no grant of its own.
|
|
18
|
-
// 3. --merge: create/refresh the three Merge rooms, borrowing this member's columns
|
|
19
|
-
// 4. grant the member SELECT on whichever Merge rooms exist — a Merge reduces to the
|
|
20
|
-
// rooms the caller can already read, so this is what makes the team room fail
|
|
21
|
-
// CLOSED rather than deny outright
|
|
22
|
-
//
|
|
23
|
-
// Grants are explicit, per room, and NEVER `ON mem.*` — a member who can read the whole
|
|
24
|
-
// house can read every other member's rooms, which would make this a shared house with
|
|
25
|
-
// longer table names. No wildcards are used anywhere; the only pattern in the design is
|
|
26
|
-
// the Merge regex, anchored on a fixed room type.
|
|
27
|
-
//
|
|
28
|
-
// Env: MEM_URL / MEM_USER / MEM_PASSWORD / MEM_DB, falling back to MEMHOUSE_*. URL and
|
|
29
|
-
// USER are REQUIRED — see the note at cfg. MEM_DB defaults to 'mem'.
|
|
30
|
-
|
|
31
|
-
const fs = require('fs');
|
|
32
|
-
const crypto = require('crypto');
|
|
33
|
-
const path = require('path');
|
|
34
|
-
// Named, not thrown as a raw MODULE_NOT_FOUND with a require stack. This runs as a
|
|
35
|
-
// SUBPROCESS of `memhouse install`, after the admin path has already created a ClickHouse
|
|
36
|
-
// user — so its failure mode is a half-provisioned house, and "Cannot find module" plus a
|
|
37
|
-
// stack trace is the least useful thing to print at that moment. Seen for real in a
|
|
38
|
-
// checkout where `npm install` had not been run.
|
|
39
|
-
let createClient, ClickHouseLogLevel;
|
|
40
|
-
try { ({ createClient, ClickHouseLogLevel } = require('@clickhouse/client')); }
|
|
41
|
-
catch {
|
|
42
|
-
console.error("provision.js: dependency '@clickhouse/client' is not installed.");
|
|
43
|
-
console.error(` From a checkout: npm install --prefix ${path.join(__dirname, '..', '..')}`);
|
|
44
|
-
console.error(` From an npm install: ${require('./rooms').installCommand()}`);
|
|
45
|
-
process.exit(2);
|
|
46
|
-
}
|
|
47
|
-
const { ROOM_TYPES, mergeRooms, assertUsableMember, MEMBER_PRIVS, MEMBER_PROFILE, MEMBER_PROFILE_SETTINGS } = require('./rooms');
|
|
48
|
-
|
|
49
|
-
const args = process.argv.slice(2);
|
|
50
|
-
const flag = (n) => args.includes(`--${n}`);
|
|
51
|
-
const opt = (n, d = null) => { const i = args.indexOf(`--${n}`); return i >= 0 ? args[i + 1] : d; };
|
|
52
|
-
|
|
53
|
-
// Defaults MUST match what the shipper and CLI use, or an operator who overrides only
|
|
54
|
-
// the thing that differs — a password, say — silently provisions rooms in a house the
|
|
55
|
-
// product never reads, or fails to authenticate as a user that does not exist.
|
|
56
|
-
//
|
|
57
|
-
// Which is why there is no default URL or user any more, here either. This file CREATES
|
|
58
|
-
// USERS AND GRANTS; pointing it at a guessed http://localhost:8123 as memhouse_root aims
|
|
59
|
-
// the most privileged operation in the product at whatever house happens to be listening.
|
|
60
|
-
// Observed: running it with no env at all made an authentication attempt against the
|
|
61
|
-
// pilot's own ClickHouse.
|
|
62
|
-
const cfg = {
|
|
63
|
-
url: process.env.MEM_URL || process.env.MEMHOUSE_URL,
|
|
64
|
-
username: process.env.MEM_USER || process.env.MEMHOUSE_USER,
|
|
65
|
-
password: process.env.MEM_PASSWORD || process.env.MEMHOUSE_PASSWORD || '',
|
|
66
|
-
database: process.env.MEM_DB || process.env.MEMHOUSE_DB || 'mem',
|
|
67
|
-
};
|
|
68
|
-
if (!cfg.url || !cfg.username) {
|
|
69
|
-
console.error('provision.js: no house given. Set MEM_URL and MEM_USER (or MEMHOUSE_URL/MEMHOUSE_USER).');
|
|
70
|
-
console.error(' This creates users and grants — it will not guess http://localhost:8123 as memhouse_root.');
|
|
71
|
-
process.exit(2);
|
|
72
|
-
}
|
|
73
|
-
|
|
74
|
-
// Statements are split on ';' at end of line — the templates contain no ';' inside a
|
|
75
|
-
// statement body, and keeping the splitter dumb keeps the templates readable.
|
|
76
|
-
function statements(sql) {
|
|
77
|
-
return sql
|
|
78
|
-
.split(/;\s*$/m)
|
|
79
|
-
.map((s) => s.replace(/^\s*--.*$/gm, '').trim())
|
|
80
|
-
.filter(Boolean);
|
|
81
|
-
}
|
|
82
|
-
|
|
83
|
-
/** Which of the three Merge rooms actually exist — a member may be provisioned first. */
|
|
84
|
-
async function mergeRoomsPresent(client, database) {
|
|
85
|
-
const names = Object.values(mergeRooms());
|
|
86
|
-
const rs = await client.query({
|
|
87
|
-
query: `SELECT name FROM system.tables WHERE database = {db:String} AND name IN ({names:Array(String)}) ORDER BY name`,
|
|
88
|
-
query_params: { db: database, names },
|
|
89
|
-
format: 'JSONEachRow',
|
|
90
|
-
});
|
|
91
|
-
return (await rs.json()).map((r) => r.name);
|
|
92
|
-
}
|
|
93
|
-
|
|
94
|
-
async function main() {
|
|
95
|
-
const member = opt('member');
|
|
96
|
-
if (!member) { console.error('usage: provision.js --member <name> [--merge]'); process.exit(2); }
|
|
97
|
-
assertUsableMember(member);
|
|
98
|
-
|
|
99
|
-
const here = __dirname;
|
|
100
|
-
|
|
101
|
-
// Create the house with a client that has NOT selected it — on a fresh server the
|
|
102
|
-
// database does not exist yet, and selecting it fails before it can be created.
|
|
103
|
-
// See the note in shipper/ship.js: the driver's ERROR-level dump buries every
|
|
104
|
-
// refusal this file prints. MEMHOUSE_DEBUG=1 restores it.
|
|
105
|
-
const quiet = { level: process.env.MEMHOUSE_DEBUG ? ClickHouseLogLevel.DEBUG : ClickHouseLogLevel.OFF };
|
|
106
|
-
const bootstrap = createClient({ ...cfg, database: '', log: quiet });
|
|
107
|
-
await bootstrap.command({ query: `CREATE DATABASE IF NOT EXISTS ${cfg.database}` });
|
|
108
|
-
await bootstrap.close();
|
|
109
|
-
|
|
110
|
-
console.log(`[mem] provisioning in ${cfg.url} database '${cfg.database}' as '${cfg.username}'`);
|
|
111
|
-
const client = createClient({ ...cfg, log: quiet, clickhouse_settings: { async_insert: 0 } });
|
|
112
|
-
|
|
113
|
-
// 0. the member themselves.
|
|
114
|
-
//
|
|
115
|
-
// This file never created the ClickHouse user, so the path `memhouse install` prints and
|
|
116
|
-
// INSTALL.md repeats — `provision.js --member <name> --merge` — created three rooms and
|
|
117
|
-
// then died on `There is no role \`<name>\` in \`user directories\``, leaving orphan rooms
|
|
118
|
-
// that nothing removes and that the --merge step then folded into all_sessions. It also
|
|
119
|
-
// ignored --member-password. The documented second-member path did not work at all.
|
|
120
|
-
const existing = await client.query({
|
|
121
|
-
query: `SELECT name FROM system.users WHERE name = {n:String}`,
|
|
122
|
-
query_params: { n: member }, format: 'JSONEachRow',
|
|
123
|
-
});
|
|
124
|
-
if ((await existing.json()).length === 0) {
|
|
125
|
-
const pw = opt('member-password') || crypto.randomBytes(24).toString('base64url');
|
|
126
|
-
await client.command({ query: `CREATE USER ${member} IDENTIFIED BY '${pw.replace(/'/g, "\\'")}'` });
|
|
127
|
-
console.log(`[mem] created ClickHouse user '${member}'`);
|
|
128
|
-
if (!opt('member-password')) {
|
|
129
|
-
console.log(`[mem] password for '${member}': ${pw}`);
|
|
130
|
-
console.log('[mem] shown once — hand it over, or pass --member-password next time');
|
|
131
|
-
}
|
|
132
|
-
console.log(`[mem] they finish with: memhouse install --url ${cfg.url} --db ${cfg.database} --user ${member} --password '…'`);
|
|
133
|
-
} else if (flag('user-just-created')) {
|
|
134
|
-
// The caller made it moments ago and has already said so. Saying "already exists"
|
|
135
|
-
// here contradicts the line above it, and does so using the exact phrase that
|
|
136
|
-
// elsewhere means "someone else owns this handle, refusing" — which reads as a
|
|
137
|
-
// warning in the middle of a successful first install.
|
|
138
|
-
console.log(`[mem] using the ClickHouse user '${member}' just created — provisioning rooms`);
|
|
139
|
-
} else {
|
|
140
|
-
// NOT "already exists". That phrase is what the identity-takeover REFUSAL says, so
|
|
141
|
-
// reading it here — during a successful provision, and especially while resuming a
|
|
142
|
-
// first install that failed after CREATE USER — reads as a warning that someone else
|
|
143
|
-
// owns the handle. Both branches of this if now describe a success, in plain words.
|
|
144
|
-
console.log(`[mem] using the existing ClickHouse user '${member}' — provisioning rooms`);
|
|
145
|
-
}
|
|
146
|
-
|
|
147
|
-
// 1. rooms
|
|
148
|
-
const tpl = fs.readFileSync(path.join(here, 'schema-member.sql.tpl'), 'utf-8');
|
|
149
|
-
const roomSql = statements(tpl.replaceAll('{{MEMBER}}', member));
|
|
150
|
-
for (const q of roomSql) {
|
|
151
|
-
// allow_experimental_full_text_index: a no-op on 26.x, required on 25.x for the
|
|
152
|
-
// messages text indexes.
|
|
153
|
-
await client.command({ query: q, clickhouse_settings: { allow_experimental_full_text_index: 1 } });
|
|
154
|
-
}
|
|
155
|
-
// CREATE TABLE IF NOT EXISTS is a no-op on an existing room, so re-running provision.js
|
|
156
|
-
// over a room that has DRIFTED left it lossy while printing "rooms ready". The owner's
|
|
157
|
-
// route has to heal as well as create — it is the one route that always holds the rights.
|
|
158
|
-
{
|
|
159
|
-
const { templateColumns } = require('../shipper/ship');
|
|
160
|
-
const want = templateColumns(tpl);
|
|
161
|
-
for (const ty of ROOM_TYPES) {
|
|
162
|
-
const room = `${ty}_${member}`;
|
|
163
|
-
const rs = await client.query({
|
|
164
|
-
query: 'SELECT name FROM system.columns WHERE database = {d:String} AND table = {n:String}',
|
|
165
|
-
query_params: { d: cfg.database, n: room }, format: 'JSONEachRow',
|
|
166
|
-
});
|
|
167
|
-
const have = new Set((await rs.json()).map((r) => r.name));
|
|
168
|
-
if (!have.size) continue;
|
|
169
|
-
for (const c of (want[ty] || [])) {
|
|
170
|
-
if (have.has(c.name)) continue;
|
|
171
|
-
await client.command({
|
|
172
|
-
query: `ALTER TABLE ${cfg.database}.${room} ADD COLUMN IF NOT EXISTS ${c.name} ${c.type}`,
|
|
173
|
-
clickhouse_settings: { allow_experimental_full_text_index: 1 },
|
|
174
|
-
});
|
|
175
|
-
console.log(`[mem] added missing column ${room}.${c.name}`);
|
|
176
|
-
}
|
|
177
|
-
}
|
|
178
|
-
}
|
|
179
|
-
console.log(`[mem] rooms ready for '${member}': ${ROOM_TYPES.map((t) => `${t}_${member}`).join(', ')}`);
|
|
180
|
-
|
|
181
|
-
// 2. grants — two statements per room, and the split is the point.
|
|
182
|
-
//
|
|
183
|
-
// ALL, NOT re-grantable: the member owns their room outright, DROP and TRUNCATE
|
|
184
|
-
// included, because it is their memory to destroy. It also covers the mutation
|
|
185
|
-
// privileges the shipper needs without naming them — the shipper clears a session's
|
|
186
|
-
// rows before re-inserting, and which of ALTER UPDATE / ALTER DELETE a server demands
|
|
187
|
-
// varies by version (`DELETE FROM` is a lightweight delete, implemented as
|
|
188
|
-
// `ALTER TABLE … UPDATE _row_exists = 0`, so 25.11 wants ALTER UPDATE where 26.7 wanted
|
|
189
|
-
// ALTER DELETE). Granting one and not the other produces a pass that fails elsewhere.
|
|
190
|
-
//
|
|
191
|
-
// SELECT, re-grantable: a share is READ-ONLY BY CONSTRUCTION rather than by convention.
|
|
192
|
-
// `GRANT ALL … WITH GRANT OPTION` expands to 45 privileges on 26.7 — DROP TABLE,
|
|
193
|
-
// TRUNCATE, CREATE ROW POLICY, SYSTEM DROP REPLICA among them — and every one of those
|
|
194
|
-
// would become something a member could hand to a colleague while meaning "let them
|
|
195
|
-
// read my sessions". Measured: with this split, `GRANT SELECT … TO bob` succeeds and
|
|
196
|
-
// `GRANT DROP TABLE … TO bob` is refused 497.
|
|
197
|
-
// NOT `GRANT ALL`. On 26.7 that expands to 45 privileges including CREATE TABLE **on
|
|
198
|
-
// the member's own room name** — which means a member owns the NAME, not just the data,
|
|
199
|
-
// and can put any engine behind it. Measured on both versions:
|
|
200
|
-
//
|
|
201
|
-
// DROP TABLE messages_alice; -- allowed
|
|
202
|
-
// CREATE TABLE messages_alice AS all_messages
|
|
203
|
-
// ENGINE = Merge('<db>','^messages_'); -- allowed
|
|
204
|
-
//
|
|
205
|
-
// `all_messages` is Merge(db,'^messages_'), so it then contains a Merge over its own
|
|
206
|
-
// namespace and every OTHER member's rows are counted twice in the team room: a house
|
|
207
|
-
// reading alice 6030 / bob 2401 / carol 12 became bob 4802 / carol 24, silently, no
|
|
208
|
-
// error. Confidentiality survives — the Merge still narrows by grants, so alice reads
|
|
209
|
-
// nothing new — but integrity does not, and a team-wide number is the whole point of
|
|
210
|
-
// the room. SCHEMA.md already records the same failure reached by accident; this is the
|
|
211
|
-
// same failure reachable on purpose by any member.
|
|
212
|
-
//
|
|
213
|
-
// So: the privileges the shipper actually uses, and nothing that lets a member define
|
|
214
|
-
// an object. ALTER UPDATE and ALTER DELETE both, because `DELETE FROM` is a lightweight
|
|
215
|
-
// delete implemented as `ALTER TABLE … UPDATE _row_exists = 0` and which of the two a
|
|
216
|
-
// server demands varies by version (25.11 wants UPDATE where 26.7 wanted DELETE).
|
|
217
|
-
// ALTER ADD COLUMN for ensureSchema's rollout. No CREATE TABLE, no DROP TABLE, no
|
|
218
|
-
// TRUNCATE: the admin who provisioned the room is the one who can replace it.
|
|
219
|
-
//
|
|
220
|
-
// SELECT is re-grantable so a share is READ-ONLY BY CONSTRUCTION rather than by
|
|
221
|
-
// convention — `GRANT SELECT … TO bob` succeeds, `GRANT DROP TABLE … TO bob` is 497.
|
|
222
|
-
for (const t of ROOM_TYPES) {
|
|
223
|
-
const room = `${cfg.database}.${t}_${member}`;
|
|
224
|
-
await client.command({ query: `GRANT ${MEMBER_PRIVS} ON ${room} TO ${member}` });
|
|
225
|
-
await client.command({ query: `GRANT SELECT ON ${room} TO ${member} WITH GRANT OPTION` });
|
|
226
|
-
}
|
|
227
|
-
console.log(`[mem] granted ${MEMBER_PRIVS} on 3 rooms to '${member}'; SELECT is the only re-grantable one`);
|
|
228
|
-
|
|
229
|
-
// 2b. A settings profile with CEILINGS, not just defaults.
|
|
230
|
-
//
|
|
231
|
-
// Every isolation test passed on confidentiality and none existed for availability: a
|
|
232
|
-
// member could `SETTINGS max_memory_usage=100000000000`, `max_execution_time=0` and
|
|
233
|
-
// `max_threads=64` on a shared house, and take the server down for everyone. The
|
|
234
|
-
// constraint form (`MAX`) is what makes it a ceiling — a plain default is advisory and a
|
|
235
|
-
// member simply overrides it, which is what they were doing.
|
|
236
|
-
//
|
|
237
|
-
// Generous on purpose: a full re-ship of a large house is a big INSERT, and the point is
|
|
238
|
-
// to stop one member exhausting the box, not to make honest work fail.
|
|
239
|
-
// CREATE ... OR REPLACE, which this used, DETACHES every user already assigned to the
|
|
240
|
-
// profile. One shared profile plus a replace on every provision meant that adding a
|
|
241
|
-
// second member silently stripped the first member's ceilings — measured: provision
|
|
242
|
-
// kyle, then provision lena, and `SHOW CREATE USER kyle` no longer carries the profile
|
|
243
|
-
// at all. In a three-member house only the most recently provisioned member had any
|
|
244
|
-
// limit, and nothing said so.
|
|
245
|
-
//
|
|
246
|
-
// CREATE IF NOT EXISTS + ALTER updates the same object in place and keeps every existing
|
|
247
|
-
// assignment, so one provision now brings ALL members up to the current settings —
|
|
248
|
-
// which is also what makes the async_insert pin reach members provisioned earlier.
|
|
249
|
-
// SETTINGS NONE first, because ALTER only upserts what it lists: a setting dropped from
|
|
250
|
-
// MEMBER_PROFILE_SETTINGS in a later version would otherwise stay enforced forever on
|
|
251
|
-
// every upgraded house, and anything an operator had added to a profile of the same name
|
|
252
|
-
// would survive too. Clearing then applying gives the same absolute state OR REPLACE had,
|
|
253
|
-
// without its detachment.
|
|
254
|
-
//
|
|
255
|
-
// The gap between the two statements is admin-only and sub-second — but it is not
|
|
256
|
-
// self-healing: a crash or a dropped connection between them leaves the profile EMPTY,
|
|
257
|
-
// so every member is unbounded until someone provisions again. Nothing here notices, and
|
|
258
|
-
// `doctor` does not check the profile. Re-run provision if one dies mid-way.
|
|
259
|
-
try {
|
|
260
|
-
await client.command({ query: `CREATE SETTINGS PROFILE IF NOT EXISTS ${MEMBER_PROFILE}` });
|
|
261
|
-
await client.command({ query: `ALTER SETTINGS PROFILE ${MEMBER_PROFILE} SETTINGS NONE` });
|
|
262
|
-
await client.command({ query: `ALTER SETTINGS PROFILE ${MEMBER_PROFILE} SETTINGS ${MEMBER_PROFILE_SETTINGS}` });
|
|
263
|
-
await client.command({ query: `ALTER USER ${member} SETTINGS PROFILE '${MEMBER_PROFILE}'` });
|
|
264
|
-
console.log(`[mem] settings profile '${MEMBER_PROFILE}' applied to '${member}' (memory, time and thread ceilings)`);
|
|
265
|
-
} catch (e) {
|
|
266
|
-
// A server where the admin cannot create profiles still gets a working member; say so
|
|
267
|
-
// rather than failing the provision.
|
|
268
|
-
console.log(`[mem] note: could not apply the '${MEMBER_PROFILE}' settings profile — ${e.message}`);
|
|
269
|
-
console.log('[mem] the member works, but nothing bounds their query resources on this server');
|
|
270
|
-
}
|
|
271
|
-
|
|
272
|
-
// 3. Merge rooms
|
|
273
|
-
if (flag('merge')) {
|
|
274
|
-
const mtpl = fs.readFileSync(path.join(here, 'schema-merge.sql.tpl'), 'utf-8');
|
|
275
|
-
for (const q of statements(mtpl.replaceAll('{{TEMPLATE_MEMBER}}', member))) {
|
|
276
|
-
await client.command({ query: q });
|
|
277
|
-
}
|
|
278
|
-
console.log('[mem] merge rooms ready: all_sessions, all_messages, all_tool_calls');
|
|
279
|
-
}
|
|
280
|
-
|
|
281
|
-
// 4. Merge-room grants. Broad on purpose, and safe for the reason the whole layout
|
|
282
|
-
// rests on: a Merge table reduces to the underlying rooms the CALLER holds grants for.
|
|
283
|
-
// Measured on 25.11 — bob with SELECT on all_messages and nothing on messages_alice
|
|
284
|
-
// reads his own row and none of alice's, with `_table` showing only messages_bob, and
|
|
285
|
-
// no error. Withholding this grant does not make anything safer; it makes the team
|
|
286
|
-
// room deny outright instead of failing closed, which is a different and worse thing.
|
|
287
|
-
const merged = await mergeRoomsPresent(client, cfg.database);
|
|
288
|
-
for (const room of merged) {
|
|
289
|
-
await client.command({ query: `GRANT SELECT ON ${cfg.database}.${room} TO ${member}` });
|
|
290
|
-
}
|
|
291
|
-
if (merged.length) console.log(`[mem] granted SELECT on ${merged.join(', ')} to '${member}'`);
|
|
292
|
-
else console.log('[mem] no merge rooms yet — run once with --merge to create them');
|
|
293
|
-
|
|
294
|
-
await client.close();
|
|
295
|
-
}
|
|
296
|
-
|
|
297
|
-
main().catch((e) => { console.error(`[mem] provision failed: ${e.message}`); process.exit(1); });
|