memhouse 0.7.1 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +29 -11
- package/bin/memhouse.js +129 -122
- package/memhouse/DESIGN.md +29 -30
- package/memhouse/delivery/PROMPT.md +7 -9
- package/memhouse/delivery/kernel-install.md +14 -29
- package/memhouse/delivery/plugin/skills/search/SKILL.md +25 -38
- package/memhouse/delivery/plugin/skills/sessions/SKILL.md +15 -28
- package/memhouse/delivery/plugin/skills/sql/SKILL.md +23 -36
- package/memhouse/host.js +1 -1
- package/memhouse/house/HOUSE.md +74 -0
- package/memhouse/house/house.js +165 -0
- package/memhouse/{per-member/schema-member.sql.tpl → house/schema.sql.tpl} +8 -11
- package/memhouse/resume.js +1 -1
- package/memhouse/server/queries.js +8 -2
- package/memhouse/shipper/ship.js +24 -30
- package/package.json +1 -1
- package/memhouse/per-member/INSTALL.md +0 -228
- package/memhouse/per-member/PLAN.md +0 -174
- package/memhouse/per-member/PROVISIONING.md +0 -177
- package/memhouse/per-member/README.md +0 -48
- package/memhouse/per-member/SCHEMA.md +0 -120
- package/memhouse/per-member/SHARING.md +0 -149
- package/memhouse/per-member/provision.js +0 -297
- package/memhouse/per-member/rooms.js +0 -252
- package/memhouse/per-member/schema-merge.sql.tpl +0 -24
|
@@ -0,0 +1,165 @@
|
|
|
1
|
+
// The house.
|
|
2
|
+
//
|
|
3
|
+
// A house is a ClickHouse DATABASE — any database, named whatever its people name it
|
|
4
|
+
// (`polat`, `team_a`, even `default`). Its rooms are three plain tables: `sessions`,
|
|
5
|
+
// `messages`, `tool_calls`. Everyone in the house writes into the SAME tables, each with
|
|
6
|
+
// their own credential, and two columns say who and where every row came from:
|
|
7
|
+
//
|
|
8
|
+
// user_id String MATERIALIZED currentUser() — stamped by the server, unforgeable by
|
|
9
|
+
// clients (async_insert is pinned to 0
|
|
10
|
+
// on the user so the stamp cannot be
|
|
11
|
+
// skipped)
|
|
12
|
+
// host LowCardinality(String) — the machine's fingerprint, minted once
|
|
13
|
+
// per install (../host.js)
|
|
14
|
+
//
|
|
15
|
+
// That pair IS the provenance model. Alice and Bob point their shippers at `team_a` and
|
|
16
|
+
// both write; `WHERE user_id = 'alice'` is one person, `WHERE host = '…'` is one machine.
|
|
17
|
+
// The model is collaborative — housemates trust each other with the house — and the
|
|
18
|
+
// boundary between houses is the database: joining someone's ClickHouse takes a database
|
|
19
|
+
// and a credential (`GRANT ALL ON team_a.* TO alice`), nothing else, because ALL on your
|
|
20
|
+
// own database reaches nothing outside it.
|
|
21
|
+
//
|
|
22
|
+
// THIS REPLACES THE PER-MEMBER LAYOUT. Rooms were `sessions_<member>` with per-member
|
|
23
|
+
// grants, Merge rooms for team reads, and isolation between members of one database.
|
|
24
|
+
// All of it is gone: suffixed names made every client resolve its room names first (and
|
|
25
|
+
// `FROM messages` a documented trap), the Merge rooms existed only to undo the
|
|
26
|
+
// splitting, and per-member isolation inside a shared house solved an adversarial
|
|
27
|
+
// problem the product does not have. Separate houses isolate; one house shares.
|
|
28
|
+
//
|
|
29
|
+
// `sessions_v` IS A SAVED QUERY, NOT AN OBJECT — substituted into the same `FROM … AS c`
|
|
30
|
+
// position a view name would occupy, running under the caller's own credential. It was
|
|
31
|
+
// briefly a stored view; that cost an object to provision and grant, and a name for the
|
|
32
|
+
// old Merge selectors to swallow.
|
|
33
|
+
|
|
34
|
+
const ROOM_TYPES = ['sessions', 'messages', 'tool_calls'];
|
|
35
|
+
|
|
36
|
+
// The one server-side setting a member carries, applied directly on the user with
|
|
37
|
+
// ADD SETTING (bare `ALTER USER … SETTINGS` REPLACES the user's whole list — measured, it
|
|
38
|
+
// wiped an operator-set ceiling; ADD SETTING merges and upserts). There is no settings
|
|
39
|
+
// profile: the shared object detached members on replace and its server-global name made
|
|
40
|
+
// two houses on one server rewrite each other's.
|
|
41
|
+
//
|
|
42
|
+
// The pin is correctness, not policy. `user_id MATERIALIZED currentUser()` is computed
|
|
43
|
+
// during the INSERT, and an ASYNC insert flushes outside that context — measured on
|
|
44
|
+
// 25.11.9.34, `async_insert=1` stores user_id as the EMPTY STRING while a sync insert
|
|
45
|
+
// stamps correctly. The shipper always passes async_insert=0; the pin guards every OTHER
|
|
46
|
+
// holder of the credential. CONST refuses the override; a client that never mentions the
|
|
47
|
+
// setting is held at 0 and its insert lands stamped.
|
|
48
|
+
const MEMBER_PIN = 'async_insert = 0 CONST';
|
|
49
|
+
|
|
50
|
+
/**
|
|
51
|
+
* The global-install command that will actually work HERE.
|
|
52
|
+
*
|
|
53
|
+
* Every site that printed `npm install -g memhouse --allow-scripts=better-sqlite3` printed
|
|
54
|
+
* a line that fails with EACCES on any distro-packaged Node, because the global prefix is
|
|
55
|
+
* root-owned — measured on a clean Ubuntu machine, exit 243. `sudo` is right for exactly
|
|
56
|
+
* one of the two causes and makes the other worse, and what tells them apart is who owns
|
|
57
|
+
* the prefix — so look, rather than guess.
|
|
58
|
+
*/
|
|
59
|
+
function installCommand() {
|
|
60
|
+
const base = 'npm install -g memhouse --allow-scripts=better-sqlite3';
|
|
61
|
+
try {
|
|
62
|
+
const { execFileSync } = require('child_process');
|
|
63
|
+
const prefix = execFileSync('npm', ['prefix', '-g'], { encoding: 'utf-8', stdio: ['ignore', 'pipe', 'ignore'] }).trim();
|
|
64
|
+
const st = require('fs').statSync(prefix);
|
|
65
|
+
if (st.uid !== process.getuid()) return `sudo ${base}`;
|
|
66
|
+
} catch { /* npm not on PATH, or an unreadable prefix: fall through to the plain form */ }
|
|
67
|
+
return base;
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
// Identifier check for anything we splice into DDL unquoted — house (database) names and
|
|
71
|
+
// user names. Refuse what would need quoting rather than quote it: a name a person cannot
|
|
72
|
+
// type bare into their own SQL is a name they will get wrong. `default` is deliberately
|
|
73
|
+
// ALLOWED as a house name — a stock container's default database is a real place to keep
|
|
74
|
+
// a house, and the pilot may pick user 'alice' with house 'default'. Only the names that
|
|
75
|
+
// are not a database at all are refused.
|
|
76
|
+
const RESERVED_DBS = new Set(['system', 'information_schema']);
|
|
77
|
+
function assertUsableName(name, what = 'name') {
|
|
78
|
+
if (typeof name !== 'string' || !/^[A-Za-z][A-Za-z0-9_]*$/.test(name)) {
|
|
79
|
+
throw new Error(`'${name}' cannot be a ${what} — expected [A-Za-z][A-Za-z0-9_]*`);
|
|
80
|
+
}
|
|
81
|
+
if (RESERVED_DBS.has(name.toLowerCase())) {
|
|
82
|
+
throw new Error(`'${name}' cannot be a ${what} — it is ClickHouse's own`);
|
|
83
|
+
}
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
/**
|
|
87
|
+
* The session rollup as SQL text, usable in the same `FROM ... AS c` position a view name
|
|
88
|
+
* would occupy. Groups by (session_id, user_id): two housemates' rows never merge, even
|
|
89
|
+
* on a colliding session_id.
|
|
90
|
+
*/
|
|
91
|
+
function sessionsRollup({ sessions, messages }) {
|
|
92
|
+
return `(
|
|
93
|
+
SELECT
|
|
94
|
+
s.session_id AS session_id,
|
|
95
|
+
any(s.source) AS source,
|
|
96
|
+
any(s.host) AS host,
|
|
97
|
+
any(s.name) AS name,
|
|
98
|
+
any(s.mode) AS mode,
|
|
99
|
+
any(s.folder) AS folder,
|
|
100
|
+
any(s.project) AS project,
|
|
101
|
+
any(s.git_branch) AS git_branch,
|
|
102
|
+
s.user_id AS user_id,
|
|
103
|
+
any(s.created_at) AS created_at,
|
|
104
|
+
any(s.last_updated_at) AS last_updated_at,
|
|
105
|
+
min(m.ts) AS started,
|
|
106
|
+
max(m.ts) AS ended,
|
|
107
|
+
coalesce(dateDiff('second', min(m.ts), max(m.ts)), 0) AS duration_sec,
|
|
108
|
+
count(m.seq) AS total_msgs,
|
|
109
|
+
countIf(m.role = 'user') AS user_msgs,
|
|
110
|
+
countIf(m.role = 'assistant') AS assistant_msgs,
|
|
111
|
+
countIf(m.is_subagent) AS subagent_msgs,
|
|
112
|
+
groupUniqArrayIf(m.model, m.model NOT IN ('', '<synthetic>')) AS models,
|
|
113
|
+
coalesce(sum(m.input_tokens), 0) AS input_tokens,
|
|
114
|
+
coalesce(sum(m.output_tokens), 0) AS output_tokens,
|
|
115
|
+
coalesce(sum(m.cache_read_tokens), 0) AS cache_read_tokens,
|
|
116
|
+
coalesce(sum(m.cache_write_tokens), 0) AS cache_write_tokens,
|
|
117
|
+
coalesce(sumIf(length(m.text), m.role = 'user'), 0) AS user_chars,
|
|
118
|
+
coalesce(sumIf(length(m.text), m.role = 'assistant'), 0) AS assistant_chars,
|
|
119
|
+
coalesce(substring(argMinIf(m.text, m.seq, m.role = 'user' AND m.text != ''), 1, 200), '') AS first_prompt
|
|
120
|
+
FROM ${sessions} AS s FINAL
|
|
121
|
+
LEFT JOIN ${messages} AS m FINAL ON m.session_id = s.session_id AND m.user_id = s.user_id
|
|
122
|
+
GROUP BY s.session_id, s.user_id
|
|
123
|
+
SETTINGS join_use_nulls = 1
|
|
124
|
+
)`;
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
/**
|
|
128
|
+
* The settings every read needs. `final` collapses ReplacingMergeTree versions;
|
|
129
|
+
* `join_use_nulls` is what the rollup's coalesce depends on.
|
|
130
|
+
*/
|
|
131
|
+
const READ_SETTINGS = { final: 1, join_use_nulls: 1 };
|
|
132
|
+
|
|
133
|
+
/**
|
|
134
|
+
* Room names. They are constants now — `sessions`, `messages`, `tool_calls`, resolved by
|
|
135
|
+
* the connection's database rather than by who is asking — but the shape survives from
|
|
136
|
+
* the per-member layout so every consumer keeps addressing rooms the same way, and
|
|
137
|
+
* `user` still carries the server-reported identity: writers need the VALUE, not the
|
|
138
|
+
* name it produces. See ship.js's delete, which must BIND the user rather than call
|
|
139
|
+
* currentUser() inside a mutation, where it is not evaluated in the caller's context.
|
|
140
|
+
*/
|
|
141
|
+
function roomNames(user) {
|
|
142
|
+
const out = { member: user, user };
|
|
143
|
+
for (const t of ROOM_TYPES) out[t] = t;
|
|
144
|
+
out.sessions_v = sessionsRollup(out);
|
|
145
|
+
return out;
|
|
146
|
+
}
|
|
147
|
+
|
|
148
|
+
async function currentUser(client) {
|
|
149
|
+
const rs = await client.query({ query: 'SELECT currentUser() AS u', format: 'JSONEachRow' });
|
|
150
|
+
const rows = await rs.json();
|
|
151
|
+
const u = rows[0] && rows[0].u;
|
|
152
|
+
if (!u) throw new Error('could not determine currentUser()');
|
|
153
|
+
return u;
|
|
154
|
+
}
|
|
155
|
+
|
|
156
|
+
/** Resolve the rooms for whoever this client is connected as. */
|
|
157
|
+
async function resolveRooms(client) {
|
|
158
|
+
return roomNames(await currentUser(client));
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
module.exports = {
|
|
162
|
+
ROOM_TYPES, READ_SETTINGS, MEMBER_PIN,
|
|
163
|
+
installCommand, assertUsableName,
|
|
164
|
+
sessionsRollup, roomNames, currentUser, resolveRooms,
|
|
165
|
+
};
|
|
@@ -1,16 +1,13 @@
|
|
|
1
|
-
--
|
|
2
|
-
--
|
|
3
|
-
--
|
|
4
|
-
--
|
|
5
|
-
--
|
|
6
|
-
-- user_id MATERIALIZED currentUser() is retained even though the room names the member:
|
|
7
|
-
-- it keeps provenance across a share and keeps the Merge rooms meaningful. Writers must
|
|
8
|
-
-- use async_insert=0 or the stamp does not happen.
|
|
1
|
+
-- The house's three rooms — plain, shared tables. Everyone in the house writes into
|
|
2
|
+
-- these same tables with their own credential; `user_id MATERIALIZED currentUser()`
|
|
3
|
+
-- (server-stamped, async_insert pinned to 0 on the user) says who, and `host` says
|
|
4
|
+
-- which machine. There is no per-member table and no Merge room: the database IS the
|
|
5
|
+
-- boundary, and a housemate is anyone granted on it.
|
|
9
6
|
--
|
|
10
7
|
-- ClickHouse >= 26.2 natively — 25.11 works because the shipper passes
|
|
11
8
|
-- allow_experimental_full_text_index=1 per-query when applying this file.
|
|
12
9
|
|
|
13
|
-
CREATE TABLE IF NOT EXISTS
|
|
10
|
+
CREATE TABLE IF NOT EXISTS sessions
|
|
14
11
|
(
|
|
15
12
|
session_id String,
|
|
16
13
|
source LowCardinality(String),
|
|
@@ -47,7 +44,7 @@ CREATE TABLE IF NOT EXISTS sessions_{{MEMBER}}
|
|
|
47
44
|
ENGINE = ReplacingMergeTree(ingested_at)
|
|
48
45
|
ORDER BY (session_id, user_id);
|
|
49
46
|
|
|
50
|
-
CREATE TABLE IF NOT EXISTS
|
|
47
|
+
CREATE TABLE IF NOT EXISTS messages
|
|
51
48
|
(
|
|
52
49
|
session_id String,
|
|
53
50
|
seq UInt32,
|
|
@@ -88,7 +85,7 @@ CREATE TABLE IF NOT EXISTS messages_{{MEMBER}}
|
|
|
88
85
|
ENGINE = ReplacingMergeTree(ingested_at)
|
|
89
86
|
ORDER BY (session_id, user_id, origin, seq);
|
|
90
87
|
|
|
91
|
-
CREATE TABLE IF NOT EXISTS
|
|
88
|
+
CREATE TABLE IF NOT EXISTS tool_calls
|
|
92
89
|
(
|
|
93
90
|
session_id String,
|
|
94
91
|
seq UInt32,
|
package/memhouse/resume.js
CHANGED
|
@@ -26,7 +26,7 @@
|
|
|
26
26
|
// `source: 'claude-code'` on every chat it returns. This table was written against the
|
|
27
27
|
// module names first and would have refused to resume a single Claude Code session — 508 of
|
|
28
28
|
// them on the machine it was tested on. Read the values out of a real house
|
|
29
|
-
// (`SELECT DISTINCT source FROM
|
|
29
|
+
// (`SELECT DISTINCT source FROM sessions`), never off the adapter's name.
|
|
30
30
|
const RESUMERS = {
|
|
31
31
|
// claude --help, 2026-08-12: "-r, --resume [value] Resume a conversation by session ID"
|
|
32
32
|
'claude-code': {
|
|
@@ -18,13 +18,13 @@
|
|
|
18
18
|
|
|
19
19
|
const { createClient, ClickHouseLogLevel } = require('@clickhouse/client');
|
|
20
20
|
const { calculateCost, normalizeModelName } = require('../../pricing');
|
|
21
|
-
const { resolveRooms, READ_SETTINGS } = require('../
|
|
21
|
+
const { resolveRooms, READ_SETTINGS } = require('../house/house');
|
|
22
22
|
|
|
23
23
|
const config = {
|
|
24
24
|
url: process.env.MEMHOUSE_URL || 'http://localhost:8123',
|
|
25
25
|
username: process.env.MEMHOUSE_USER || 'memhouse_root',
|
|
26
26
|
password: process.env.MEMHOUSE_PASSWORD || '',
|
|
27
|
-
database: process.env.MEMHOUSE_DB || '
|
|
27
|
+
database: process.env.MEMHOUSE_DB || process.env.MEMHOUSE_USER, // house defaults to the user's own name
|
|
28
28
|
};
|
|
29
29
|
|
|
30
30
|
let _client = null;
|
|
@@ -925,6 +925,12 @@ async function rawQuery(sql) {
|
|
|
925
925
|
// Cut the result off rather than failing a query that was legitimately large.
|
|
926
926
|
result_overflow_mode: 'break',
|
|
927
927
|
max_execution_time: 30,
|
|
928
|
+
// Result caps bound what LEAVES the server; this bounds what the query does inside
|
|
929
|
+
// it — a huge GROUP BY hits none of the caps above while eating the box's memory.
|
|
930
|
+
// Used to come from the member settings profile; that object is gone (it detached
|
|
931
|
+
// members on replace and its name was server-global), so the one caller that pipes
|
|
932
|
+
// arbitrary SQL carries its own ceiling.
|
|
933
|
+
max_memory_usage: 8000000000,
|
|
928
934
|
});
|
|
929
935
|
const columns = rows.length > 0 ? Object.keys(rows[0]) : [];
|
|
930
936
|
return { columns, rows, count: rows.length, truncated: rows.length >= RAW_MAX_ROWS };
|
package/memhouse/shipper/ship.js
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
// memhouse SHIPPER — parse-on-client (bet #1 in ../DESIGN.md). Runs the 17 editor
|
|
3
|
-
// adapters locally and ships TYPED rows into
|
|
3
|
+
// adapters locally and ships TYPED rows into the house's shared rooms — sessions /
|
|
4
4
|
// messages_<m> / tool_calls_<m>, where <m> is `SELECT currentUser()` and never config
|
|
5
|
-
// (../
|
|
5
|
+
// (../house/schema.sql.tpl, ../house/house.js). All of one member's
|
|
6
6
|
// machines land in that member's rooms, told apart by the `host` column; no member
|
|
7
7
|
// writes into another's. The typed evolution of agency/ingest.js: same host id,
|
|
8
8
|
// same ts interpolation, same batching discipline — but rows land in physical
|
|
@@ -43,7 +43,7 @@ try { ({ createClient, ClickHouseLogLevel } = require('@clickhouse/client')); }
|
|
|
43
43
|
catch {
|
|
44
44
|
console.error("[memhouse] dependency '@clickhouse/client' is not installed.");
|
|
45
45
|
console.error(`[memhouse] from a checkout: npm install --prefix ${require('path').join(__dirname, '..', '..')}`);
|
|
46
|
-
console.error(`[memhouse] from an npm install: ${require('../
|
|
46
|
+
console.error(`[memhouse] from an npm install: ${require('../house/house').installCommand()}`);
|
|
47
47
|
process.exit(2);
|
|
48
48
|
}
|
|
49
49
|
const { getAllChats, getAdapterErrors, getMessages, resetCaches } = require('../../editors');
|
|
@@ -52,7 +52,7 @@ const selfUpdate = require('../self-update');
|
|
|
52
52
|
// Captured at require time, which is as close to "what this process booted with" as it
|
|
53
53
|
// gets. Taking it later would record whatever an upgrade had already replaced.
|
|
54
54
|
const selfSnap = selfUpdate.snapshot(__filename);
|
|
55
|
-
const { resolveRooms, READ_SETTINGS, ROOM_TYPES } = require('../
|
|
55
|
+
const { resolveRooms, READ_SETTINGS, ROOM_TYPES } = require('../house/house');
|
|
56
56
|
|
|
57
57
|
const BATCH_ROWS = 2000; // insert batch ceiling (binding)
|
|
58
58
|
const TEXT_MAX = 50000; // messages.text truncation
|
|
@@ -203,7 +203,7 @@ function makeClient() {
|
|
|
203
203
|
url: process.env.MEMHOUSE_URL,
|
|
204
204
|
username: process.env.MEMHOUSE_USER,
|
|
205
205
|
password: process.env.MEMHOUSE_PASSWORD || '',
|
|
206
|
-
database: process.env.MEMHOUSE_DB || '
|
|
206
|
+
database: process.env.MEMHOUSE_DB || process.env.MEMHOUSE_USER, // house defaults to the user's own name
|
|
207
207
|
request_timeout: 300000, // full re-ships move tens of MB; don't cut inserts short
|
|
208
208
|
clickhouse_settings: {
|
|
209
209
|
// Int64/UInt64 back as JSON numbers — our values (counts, tokens) are < 2^53.
|
|
@@ -221,7 +221,7 @@ function makeClient() {
|
|
|
221
221
|
});
|
|
222
222
|
}
|
|
223
223
|
|
|
224
|
-
// Create the
|
|
224
|
+
// Create the house's rooms, from ../house/schema.sql.tpl. Comments are
|
|
225
225
|
// stripped BEFORE the ';' split — schema comments legitimately contain semicolons.
|
|
226
226
|
//
|
|
227
227
|
// This is the solo path: on a house you own, `memhouse install` mints your three rooms and
|
|
@@ -229,9 +229,8 @@ function makeClient() {
|
|
|
229
229
|
// same identity the rooms' user_id is stamped with, so a client cannot name its way into
|
|
230
230
|
// someone else's rooms.
|
|
231
231
|
//
|
|
232
|
-
// It
|
|
233
|
-
//
|
|
234
|
-
// ../per-member/provision.js.
|
|
232
|
+
// It issues no grants: joining a house is the admin's two statements (CREATE USER +
|
|
233
|
+
// GRANT ALL ON the house), and once granted, each member's own shipper runs this.
|
|
235
234
|
/**
|
|
236
235
|
* Refuse to write into rooms whose sorting key is wrong for `origin` — in EITHER
|
|
237
236
|
* direction. Two different houses are broken in two opposite ways:
|
|
@@ -251,7 +250,7 @@ function makeClient() {
|
|
|
251
250
|
* Refuse, legibly, when the caller's rooms are not there.
|
|
252
251
|
*
|
|
253
252
|
* Without this the first thing to touch a roomless house is loadExisting's SELECT, and the
|
|
254
|
-
* user gets `Unknown table expression identifier '
|
|
253
|
+
* user gets `Unknown table expression identifier 'sessions' in scope SELECT session_id,
|
|
255
254
|
* last_updated_at, message_count, extra FROM ...` — a raw ClickHouse identifier error with
|
|
256
255
|
* no next step, from a command that may be running inside a service loop. `install` and
|
|
257
256
|
* `doctor` both handle the identical situation properly; ship was the one that did not.
|
|
@@ -305,7 +304,7 @@ function templateColumns(tpl) {
|
|
|
305
304
|
*/
|
|
306
305
|
async function warnMissingColumns(client, rooms) {
|
|
307
306
|
let tpl;
|
|
308
|
-
try { tpl = fs.readFileSync(path.join(__dirname, '..', '
|
|
307
|
+
try { tpl = fs.readFileSync(path.join(__dirname, '..', 'house', 'schema.sql.tpl'), 'utf-8'); }
|
|
309
308
|
catch { return; }
|
|
310
309
|
const want = templateColumns(tpl);
|
|
311
310
|
const missing = [];
|
|
@@ -375,7 +374,7 @@ async function assertOriginKeyed(client, rooms) {
|
|
|
375
374
|
+ wrong.map((s) => ` ${s}`).join('\n')
|
|
376
375
|
+ '\n ORDER BY cannot be altered in place. Rebuild each room, then re-ship:'
|
|
377
376
|
+ '\n RENAME TABLE <room> TO <room>_old;'
|
|
378
|
-
+ `\n -- recreate from ${path.join(__dirname, '..', '
|
|
377
|
+
+ `\n -- recreate from ${path.join(__dirname, '..', 'house', 'schema.sql.tpl')}`
|
|
379
378
|
+ '\n INSERT INTO <room> SELECT * FROM <room>_old;'
|
|
380
379
|
+ '\n (add `, \'ship\' AS origin` to the SELECT only if <room>_old has no origin column)'
|
|
381
380
|
+ '\n A house with no imported rows can also just be re-shipped from scratch.');
|
|
@@ -384,18 +383,19 @@ async function assertOriginKeyed(client, rooms) {
|
|
|
384
383
|
|
|
385
384
|
async function ensureSchema(client) {
|
|
386
385
|
const rooms = await resolveRooms(client);
|
|
387
|
-
const
|
|
388
|
-
const
|
|
389
|
-
const sql = tpl.replaceAll('{{MEMBER}}', member);
|
|
386
|
+
const tpl = fs.readFileSync(path.join(__dirname, '..', 'house', 'schema.sql.tpl'), 'utf-8');
|
|
387
|
+
const sql = tpl; // plain shared tables — nothing to render
|
|
390
388
|
const stripped = sql.split('\n').map((l) => l.replace(/--.*$/, '')).join('\n');
|
|
391
389
|
const stmts = stripped.split(';').map((s) => s.trim()).filter(Boolean);
|
|
392
|
-
//
|
|
393
|
-
//
|
|
394
|
-
//
|
|
395
|
-
//
|
|
396
|
-
//
|
|
397
|
-
//
|
|
398
|
-
//
|
|
390
|
+
// Two failure shapes are survivable here and both are skipped rather than fatal:
|
|
391
|
+
//
|
|
392
|
+
// - a permission denial. A read-only credential can still run --ensure-schema for
|
|
393
|
+
// its ADD COLUMN rollout; ClickHouse checks the grant BEFORE existence, so IF NOT
|
|
394
|
+
// EXISTS does not save the CREATEs. Skip what we may not do and carry on.
|
|
395
|
+
// - a create RACE. Two housemates' first-ever ships can run these CREATEs
|
|
396
|
+
// concurrently, and IF NOT EXISTS is not atomic against a simultaneous creator —
|
|
397
|
+
// the loser can get TABLE_ALREADY_EXISTS or a metadata-file collision. The table
|
|
398
|
+
// exists either way, which is the outcome this function wants.
|
|
399
399
|
let denied = 0;
|
|
400
400
|
for (const q of stmts) {
|
|
401
401
|
try {
|
|
@@ -410,6 +410,7 @@ async function ensureSchema(client) {
|
|
|
410
410
|
} catch (e) {
|
|
411
411
|
const m = e && e.message ? e.message : String(e);
|
|
412
412
|
if (/Not enough privileges|ACCESS_DENIED/i.test(m)) { denied++; continue; }
|
|
413
|
+
if (/ALREADY_EXISTS|already exists/i.test(m)) { continue; } // lost a create race — the winner made it
|
|
413
414
|
throw e;
|
|
414
415
|
}
|
|
415
416
|
}
|
|
@@ -422,13 +423,6 @@ async function ensureSchema(client) {
|
|
|
422
423
|
// would error. Add it in place; ReplacingMergeTree backfills the DEFAULT, so every
|
|
423
424
|
// pre-existing row reads as 'ship', which is what it was.
|
|
424
425
|
//
|
|
425
|
-
// Deliberately NOT applied to the Merge rooms — but not for the reason this comment used
|
|
426
|
-
// to give. It claimed they "reject ALTER"; they do not, on either 26.7 or 25.11, where
|
|
427
|
-
// `ALTER TABLE all_messages ADD COLUMN …` succeeds. The real reason is that a Merge room
|
|
428
|
-
// is a VIEW over whatever `^messages_` matches: a column added to it is cosmetic, is not
|
|
429
|
-
// backed by the underlying rooms, and drifts from them the moment a member is added. It
|
|
430
|
-
// takes its structure from a member room at CREATE time, so recreating it is the only
|
|
431
|
-
// correct way to change it — and nothing reads `origin` through it anyway.
|
|
432
426
|
// EVERY column the template declares, not just `origin`.
|
|
433
427
|
//
|
|
434
428
|
// This used to add exactly one column, which meant any OTHER column missing from a room
|
|
@@ -807,7 +801,7 @@ function reportAdapterErrors(warned) {
|
|
|
807
801
|
const noBinding = errors.filter((e) => e.missingBinding).map((e) => e.source);
|
|
808
802
|
if (noBinding.length) {
|
|
809
803
|
console.log(`[memhouse] WARNING: ${noBinding.length} adapter(s) skipped, sessions NOT shipped — better-sqlite3 has no native binding: ${noBinding.join(', ')}`);
|
|
810
|
-
console.log(`[memhouse] fix: ${require('../
|
|
804
|
+
console.log(`[memhouse] fix: ${require('../house/house').installCommand()}`);
|
|
811
805
|
}
|
|
812
806
|
for (const e of errors.filter((x) => !x.missingBinding)) {
|
|
813
807
|
console.log(`[memhouse] WARNING: ${e.source} skipped — ${e.message}`);
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "memhouse",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.8.0",
|
|
4
4
|
"description": "memhouse — agent conversation memory across 17 editors: parse-on-client shippers, a typed ClickHouse store with one set of rooms per member, kernel-agency install, and the agentlytics dashboard",
|
|
5
5
|
"main": "bin/memhouse.js",
|
|
6
6
|
"bin": {
|
|
@@ -1,228 +0,0 @@
|
|
|
1
|
-
# INSTALL — how a house gets a member
|
|
2
|
-
|
|
3
|
-
Three options. The first two are finished and measured; the third is not.
|
|
4
|
-
|
|
5
|
-
| | you hold admin | who runs the SQL | admin credential |
|
|
6
|
-
|---|---|---|---|
|
|
7
|
-
| **1. print the SQL** | yes | you | never leaves your terminal |
|
|
8
|
-
| **2. hand it over** | yes | the installer | used, then discarded |
|
|
9
|
-
| **3. kernel** | no — the realm does | not finalized | — |
|
|
10
|
-
|
|
11
|
-
## The identity rule, first — everything else follows from it
|
|
12
|
-
|
|
13
|
-
**The ClickHouse user is the identity.** Rooms are `sessions_<member>`,
|
|
14
|
-
`messages_<member>`, `tool_calls_<member>`, and `<member>` is whatever the server answers
|
|
15
|
-
to `SELECT currentUser()` — never what the config file says, never what the OS says.
|
|
16
|
-
|
|
17
|
-
The home directory produces a **suggestion**:
|
|
18
|
-
|
|
19
|
-
```
|
|
20
|
-
/Users/polat -> polat
|
|
21
|
-
/Users/ramazan.polat -> ramazan_polat (sanitized, and the change is printed)
|
|
22
|
-
/home/JohnSmith -> JohnSmith (case preserved; ClickHouse is case-sensitive)
|
|
23
|
-
/root -> refused (a container artefact, not a person)
|
|
24
|
-
```
|
|
25
|
-
|
|
26
|
-
Sanitization is `[A-Za-z][A-Za-z0-9_]*`. It can collide — `ramazan.polat` and
|
|
27
|
-
`ramazan-polat` both land on `ramazan_polat` — so a changed suggestion is printed, and
|
|
28
|
-
`--member` overrides it.
|
|
29
|
-
|
|
30
|
-
## Option 1 — print the SQL, run it yourself
|
|
31
|
-
|
|
32
|
-
The common case: you installed ClickHouse, you have `default` with
|
|
33
|
-
`access_management = 1`, and you would rather run four kinds of statement than hand a
|
|
34
|
-
superuser credential to an installer.
|
|
35
|
-
|
|
36
|
-
```
|
|
37
|
-
memhouse install --print-sql [--member polat] [--db mem]
|
|
38
|
-
```
|
|
39
|
-
|
|
40
|
-
Prints the whole install as SQL and exits. Nothing is contacted, nothing is written. It
|
|
41
|
-
generates a password and prints it inside the `CREATE USER`, or takes `--member-password`.
|
|
42
|
-
The last line is the command to run afterwards, with the credential filled in.
|
|
43
|
-
|
|
44
|
-
Two properties that make the block safe to paste anywhere:
|
|
45
|
-
|
|
46
|
-
- **Every name is qualified and there is no `USE`.** It runs in `clickhouse-client`, the
|
|
47
|
-
play UI, `curl`, or a GUI, in any database context.
|
|
48
|
-
- **The Merge engine gets the literal house name, not `currentDatabase()`.** That call is
|
|
49
|
-
evaluated at CREATE time, not read time, so a block run without the house selected would
|
|
50
|
-
otherwise build Merge rooms pointing at `default` that return zero rows forever, with no
|
|
51
|
-
error at any point.
|
|
52
|
-
|
|
53
|
-
Measured: 18 statements (the 17 objects plus a leading `SET
|
|
54
|
-
allow_experimental_full_text_index = 1`), run over HTTP with no database selected, 18
|
|
55
|
-
succeeded on both 26.7.2.59 and 25.11.9.34, and the
|
|
56
|
-
Merge rooms came out bound to `Merge('mem', '^sessions_')`.
|
|
57
|
-
|
|
58
|
-
## Option 2 — give the installer admin, it runs the SQL
|
|
59
|
-
|
|
60
|
-
```
|
|
61
|
-
memhouse install --admin-user … --admin-password … [--member polat]
|
|
62
|
-
```
|
|
63
|
-
|
|
64
|
-
The same statements, applied by `provision.js` so there is one implementation of the grant
|
|
65
|
-
set rather than two that drift. Then the step that makes the mode trustworthy:
|
|
66
|
-
**disconnect from admin, reconnect as the member, and prove that credential reaches all
|
|
67
|
-
three rooms before writing anything.** A mode that only proves the admin could do it has
|
|
68
|
-
proven nothing about the credential it is about to persist.
|
|
69
|
-
|
|
70
|
-
**The admin credential is never persisted** — not to the env file, not to a unit file, not
|
|
71
|
-
to a log. A shipper running as the house owner would make per-member rooms decoration.
|
|
72
|
-
|
|
73
|
-
## Option 3 — the kernel
|
|
74
|
-
|
|
75
|
-
**Not finalized.** The realm holds the privileged credential and mints members; what the
|
|
76
|
-
installer asks it for, and what it gets back, is not designed yet. Everything above about
|
|
77
|
-
the room layout, the grant set and the identity rule holds regardless — what is open is
|
|
78
|
-
who plays admin and how the handle is chosen.
|
|
79
|
-
|
|
80
|
-
## The grant set — two statements per room, and the split is the point
|
|
81
|
-
|
|
82
|
-
```sql
|
|
83
|
-
GRANT SELECT, INSERT, ALTER UPDATE, ALTER DELETE, ALTER ADD COLUMN, OPTIMIZE
|
|
84
|
-
ON mem.sessions_polat TO polat;
|
|
85
|
-
GRANT SELECT ON mem.sessions_polat TO polat WITH GRANT OPTION;
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
**Exactly what the shipper uses, not `ALL`.** `ALL` expands to 45 privileges on 26.7 and
|
|
89
|
-
one of them is `CREATE TABLE` **on the member's own room name** — so the member owns the
|
|
90
|
-
NAME, not just the data, and can do this:
|
|
91
|
-
|
|
92
|
-
```sql
|
|
93
|
-
DROP TABLE messages_polat;
|
|
94
|
-
CREATE TABLE messages_polat AS all_messages ENGINE = Merge('mem','^messages_');
|
|
95
|
-
```
|
|
96
|
-
|
|
97
|
-
`all_messages` is `Merge(mem,'^messages_')`, so it then contains a Merge over its own
|
|
98
|
-
namespace and **every other member's rows are counted twice in the team room**. Measured:
|
|
99
|
-
a house reading alice 6030 / bob 2401 / carol 12 became bob 4802 / carol 24, with no error
|
|
100
|
-
on either ClickHouse version. Confidentiality survives — the Merge still narrows by grants,
|
|
101
|
-
so nothing new becomes readable — but a team-wide number is what that room is for.
|
|
102
|
-
|
|
103
|
-
`ALTER UPDATE` and `ALTER DELETE` are both named because `DELETE FROM` is a lightweight
|
|
104
|
-
delete implemented as `ALTER TABLE … UPDATE _row_exists = 0`, and which of the two a server
|
|
105
|
-
demands varies by version (25.11 wants one where 26.7 wanted the other). `ALTER ADD COLUMN`
|
|
106
|
-
is for `ensureSchema`'s rollout. Replacing a room is the admin's job, not the member's.
|
|
107
|
-
|
|
108
|
-
**`SELECT`, re-grantable.** A share is read-only **by construction**, not by convention:
|
|
109
|
-
the member can hand a colleague `SELECT` and nothing else, because they hold nothing else
|
|
110
|
-
to hand over.
|
|
111
|
-
|
|
112
|
-
Measured on 26.7.2 and 25.11.9:
|
|
113
|
-
|
|
114
|
-
```
|
|
115
|
-
polat grants SELECT to bob ok
|
|
116
|
-
polat grants DROP TABLE to bob 497 Not enough privileges
|
|
117
|
-
polat drops their own room 497 Not enough privileges
|
|
118
|
-
polat creates any table 497 Not enough privileges
|
|
119
|
-
polat installs, ships, resets ok
|
|
120
|
-
bob ends up with GRANT SELECT ON mem.sessions_polat
|
|
121
|
-
```
|
|
122
|
-
|
|
123
|
-
One definition, in `rooms.js` as `MEMBER_PRIVS`, used by both `provision.js` and
|
|
124
|
-
`install --print-sql`. They drifted once — the first was narrowed and the second went on
|
|
125
|
-
emitting `GRANT ALL` for a release, so the attack above stayed reachable through the path
|
|
126
|
-
this document calls the common case.
|
|
127
|
-
|
|
128
|
-
Never `ON mem.*`. A member who can read `mem.*` can read every other member's rooms, and
|
|
129
|
-
then this is a shared house with longer table names.
|
|
130
|
-
|
|
131
|
-
`SELECT` on the three Merge rooms is granted broadly and is safe: a Merge table reduces to
|
|
132
|
-
the underlying rooms the *caller* holds grants for, so it narrows rather than denies, and
|
|
133
|
-
picks up members added later with no DDL.
|
|
134
|
-
|
|
135
|
-
## Refusals — the complete list
|
|
136
|
-
|
|
137
|
-
Install refuses, names both sides, and creates nothing:
|
|
138
|
-
|
|
139
|
-
| Case | Why |
|
|
140
|
-
|---|---|
|
|
141
|
-
| handle does not match `[A-Za-z][A-Za-z0-9_]*` after sanitization | it would need quoting, and a quoted room name breaks the Merge selectors |
|
|
142
|
-
| handle would be `root`, or the home directory is `/root` | a container artefact, not a person |
|
|
143
|
-
| **option 2: the ClickHouse user already exists** | the second human would take over the first's identity and rooms, and `user_id MATERIALIZED currentUser()` would stamp them identically. Requires `--adopt-user` AND `--member-password`, verified by reconnecting before anything is written |
|
|
144
|
-
| option 2: `mem` absent and admin cannot `CREATE DATABASE` | nothing downstream can work |
|
|
145
|
-
| an env file already points at a different url/db | it would orphan a running shipper. `memhouse setup` is the deliberate way to move; `--force` overrides |
|
|
146
|
-
| rooms missing and you cannot create them | you are not the owner — both other options are printed |
|
|
147
|
-
|
|
148
|
-
`--adopt-user` is legitimate — the same person on a new laptop — but it is
|
|
149
|
-
indistinguishable from a takeover without the password, so the password is the proof.
|
|
150
|
-
|
|
151
|
-
## After install — when admin is needed again
|
|
152
|
-
|
|
153
|
-
- **a second member** — `provision.js --member <name> --merge`, or `--print-sql --member <name>`
|
|
154
|
-
- **schema rollout** — `ADD COLUMN` on the **Merge room first**, then the member rooms;
|
|
155
|
-
Merge rejects mutations and tolerates drift, which is what makes a progressive rollout
|
|
156
|
-
possible
|
|
157
|
-
- **offboarding** — `DROP USER <member>`. The rooms are deliberately left: they hold that
|
|
158
|
-
person's transcripts, and deleting someone's memory is a decision, not a cleanup step
|
|
159
|
-
|
|
160
|
-
## Upgrading from 0.3.x — read this before you do
|
|
161
|
-
|
|
162
|
-
**0.4.0 cannot read a 0.3.x house.** An earlier version of this document said an existing
|
|
163
|
-
house "keeps working untouched". That was wrong, and it was wrong in the release notes and
|
|
164
|
-
the `v0.4.0` tag message too.
|
|
165
|
-
|
|
166
|
-
What is true: the DATABASE NAME survives. `MEMHOUSE_DB` sits in `~/.memhouse/env` and
|
|
167
|
-
explicit config outranks the new `mem` default, so nothing repoints you at a different
|
|
168
|
-
house. What does not survive is the TABLES. 0.3.x kept `sessions`, `messages`,
|
|
169
|
-
`tool_calls` and a stored `sessions_v`; 0.4.0 looks for `sessions_<you>` and finds nothing.
|
|
170
|
-
Measured against a real 0.3.2 house:
|
|
171
|
-
|
|
172
|
-
```
|
|
173
|
-
memhouse status -> not connected: Code: 60 ... Unknown table expression
|
|
174
|
-
identifier 'sessions_memhouse_root'
|
|
175
|
-
memhouse doctor -> schema: 0/3 rooms in 'memhouse'
|
|
176
|
-
memhouse ship -> pass failed: UNKNOWN_TABLE
|
|
177
|
-
```
|
|
178
|
-
|
|
179
|
-
`install` and `doctor` now detect this and name it, rather than telling you to run an
|
|
180
|
-
install you already ran.
|
|
181
|
-
|
|
182
|
-
**Your transcripts are not lost.** memhouse ships FROM your local session stores, so the
|
|
183
|
-
new rooms rebuild from disk and the old tables are left untouched:
|
|
184
|
-
|
|
185
|
-
```
|
|
186
|
-
memhouse install --admin-user <user> --admin-password <pw> --member <you>
|
|
187
|
-
memhouse ship --full
|
|
188
|
-
```
|
|
189
|
-
|
|
190
|
-
The one thing that does not come back that way is a session whose transcript you have
|
|
191
|
-
since deleted locally — it exists only in the old tables. Read those before dropping
|
|
192
|
-
anything: `SELECT * FROM <db>.sessions`.
|
|
193
|
-
|
|
194
|
-
## Where the credential lives
|
|
195
|
-
|
|
196
|
-
`~/.memhouse/env`, mode 0600, `MEMHOUSE_*` keys. **Not** a `.env` in a working directory:
|
|
197
|
-
the skills and docs source this path (`. ~/.memhouse/env`), and a `.env` beside a checkout
|
|
198
|
-
gets committed eventually.
|
|
199
|
-
|
|
200
|
-
The env file is written **last**, after everything above has proved out. Written first, a
|
|
201
|
-
failed install leaves a config behind that the next command reads as truth.
|
|
202
|
-
|
|
203
|
-
`memhouse service install` inlines the same values into the unit file, because a service
|
|
204
|
-
must not depend on a shell-quoted file it does not parse the same way. That is a second
|
|
205
|
-
copy, also 0600, removed by `memhouse service uninstall`.
|
|
206
|
-
|
|
207
|
-
## `--adopt-user` needs house admin — it is not the new-machine path
|
|
208
|
-
|
|
209
|
-
`--adopt-user` is read only on the `--admin-user` branch, so it provisions a member and
|
|
210
|
-
requires admin on the house. Passing it without `--admin-user` used to be ignored silently;
|
|
211
|
-
it is refused now.
|
|
212
|
-
|
|
213
|
-
The actual "same person, new machine" path needs no admin at all — you already have a
|
|
214
|
-
member credential, so just use it:
|
|
215
|
-
|
|
216
|
-
memhouse install --url https://… --user <member> --password …
|
|
217
|
-
|
|
218
|
-
That finds the existing rooms and writes the config. `--adopt-user` is for the case where
|
|
219
|
-
an admin is *creating or re-attaching* a member and wants the existing password verified
|
|
220
|
-
before anything is written.
|
|
221
|
-
|
|
222
|
-
## Offboarding, from the member's side
|
|
223
|
-
|
|
224
|
-
`DROP USER <member>` leaves their rooms and revokes everything (verified: no orphan grants
|
|
225
|
-
survive). From that member's machine every command then fails with `AUTHENTICATION_FAILED`
|
|
226
|
-
and advises reinstalling — which cannot work, and is indistinguishable from a wrong
|
|
227
|
-
password. There is no signal that says "you were offboarded"; ClickHouse does not
|
|
228
|
-
distinguish the two either. Worth knowing before you use it on someone.
|