memhouse 0.17.0 → 0.18.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +96 -30
- package/bin/memhouse.js +418 -196
- package/editors/antigravity.js +2 -0
- package/editors/claude.js +3 -1
- package/editors/codebuff.js +2 -0
- package/editors/codex.js +4 -1
- package/editors/commandcode.js +2 -1
- package/editors/copilot-jetbrains.js +2 -1
- package/editors/copilot.js +2 -1
- package/editors/cursor-agent.js +2 -1
- package/editors/cursor.js +2 -0
- package/editors/gemini.js +2 -1
- package/editors/goose.js +2 -1
- package/editors/index.js +8 -1
- package/editors/kiro.js +3 -2
- package/editors/opencode.js +2 -1
- package/editors/scope.js +126 -0
- package/editors/vscode.js +2 -0
- package/editors/zed.js +2 -1
- package/memhouse/DESIGN.md +31 -8
- package/memhouse/channel.js +50 -0
- package/memhouse/delivery/AGENT-INSTALL-PROMPT.md +2 -1
- package/memhouse/delivery/AGENT-INSTALL.md +1 -1
- package/memhouse/delivery/INVITATION.md.tpl +130 -0
- package/memhouse/delivery/PROMPT.md +6 -5
- package/memhouse/delivery/kernel-install.md +4 -2
- package/memhouse/delivery/plugin/.claude-plugin/plugin.json +1 -1
- package/memhouse/delivery/plugin/reference/HOUSE.md +31 -12
- package/memhouse/delivery/plugin/skills/access/SKILL.md +8 -8
- package/memhouse/delivery/plugin/skills/admin/SKILL.md +1 -1
- package/memhouse/delivery/plugin/skills/house/SKILL.md +2 -2
- package/memhouse/flags.js +5 -4
- package/memhouse/house/HOUSE.md +25 -11
- package/memhouse/house/house.js +110 -27
- package/memhouse/house/migrate.js +7 -6
- package/memhouse/house/relocate.js +2 -2
- package/memhouse/house/schema.sql.tpl +3 -3
- package/memhouse/provision.js +100 -0
- package/memhouse/shipper/ship.js +87 -28
- package/package.json +1 -1
|
@@ -36,11 +36,12 @@ FIRST. Only say you don't know after a search comes back empty.
|
|
|
36
36
|
|
|
37
37
|
Without skills, query directly.
|
|
38
38
|
|
|
39
|
-
**Table names.** The
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
39
|
+
**Table names.** The rooms are `messages`, `sessions`, `tool_calls` in the connection's
|
|
40
|
+
database — unless this is a shared house, where they carry the member's own prefix
|
|
41
|
+
(`alice_messages`). `memhouse rooms` prints what yours are actually called; a bare
|
|
42
|
+
`FROM messages` is not a table you have in a shared house. `user_id` (server-stamped)
|
|
43
|
+
says whose row it is and `host` says which machine. To read across several members, UNION ALL over the rooms you hold — naming a room
|
|
44
|
+
you do not hold is an error rather than a silent omission. **There is no `sessions_v` object** — the rollup is a saved query, and
|
|
44
45
|
`memhouse sessions-query` prints it ready to drop into a `FROM (...) AS c`.
|
|
45
46
|
|
|
46
47
|
**One session can be stored more than once.** The shipper never deletes: when an editor
|
|
@@ -80,8 +80,10 @@ user so the stamp cannot be skipped). Her machines are told apart by `host`.
|
|
|
80
80
|
**A house is shared by its housemates.** The tables are common; `user_id` and `host` say
|
|
81
81
|
who wrote what, and `WHERE user_id = 'alice'` is one person. The boundary is the
|
|
82
82
|
DATABASE: a member of `mem` holds nothing on any other database, so two agencies on one
|
|
83
|
-
kernel ClickHouse cannot read each other.
|
|
84
|
-
|
|
83
|
+
kernel ClickHouse cannot read each other. WITHIN a house, members hold their own rooms
|
|
84
|
+
and nothing else — housemates are isolated from each other by grant, not by trust, and
|
|
85
|
+
each can verify it with `SHOW GRANTS`. A separate house remains the boundary between
|
|
86
|
+
groups that should not know of each other at all.
|
|
85
87
|
|
|
86
88
|
Row policies are still the only way to share a *subset* of rows, which is owner-mediated
|
|
87
89
|
and documented in `SHARING.md` — they are not how isolation works.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "mem",
|
|
3
3
|
"description": "Conversation memory for AI agents over the memhouse typed ClickHouse store: recall past sessions and answer from them, run analytics, check the house, bring people in, and administer the server. Skills install as /mem:recall, /mem:sql, /mem:house, /mem:access, and /mem:admin.",
|
|
4
|
-
"version": "0.
|
|
4
|
+
"version": "0.18.0",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "Ramazan Polat"
|
|
7
7
|
},
|
|
@@ -8,13 +8,20 @@ or wrong everywhere.
|
|
|
8
8
|
|
|
9
9
|
## What this is
|
|
10
10
|
|
|
11
|
-
A **house** is a ClickHouse **database
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
11
|
+
A **house** is a ClickHouse **database** — on the user's machine, their server, or a
|
|
12
|
+
hosted ClickHouse; memhouse runs no service of its own. Its **rooms** are tables —
|
|
13
|
+
`sessions`, `messages`, `tool_calls` — and the server stamps who wrote every row. A
|
|
14
|
+
house's grants reach nothing outside it.
|
|
15
15
|
|
|
16
|
-
|
|
17
|
-
|
|
16
|
+
Every member's rooms are named for them — `mem.alice_messages`, `mem.alice_sessions`,
|
|
17
|
+
`mem.alice_tool_calls` — and one grant covers exactly those: `ON mem.alice_*`. Housemates
|
|
18
|
+
keep their own rooms beside yours and cannot read them, list them, or grant them onward;
|
|
19
|
+
you cannot read theirs. `SHOW GRANTS` shows you the one line you hold. **A bare
|
|
20
|
+
`FROM messages` names nothing you have** — the `q()` below prefixes room names with your
|
|
21
|
+
member name for you, and `memhouse rooms` prints what yours are called.
|
|
22
|
+
|
|
23
|
+
An **operator** owns the ClickHouse itself; an **admin credential** is what proves it.
|
|
24
|
+
`memhouse whoami` says which you are holding.
|
|
18
25
|
|
|
19
26
|
Rows arrive from a **shipper** that parses local transcripts from 17 editors. It only
|
|
20
27
|
ever inserts. Nothing in memhouse runs an LLM over your data — these skills retrieve
|
|
@@ -45,7 +52,8 @@ if [ -z "${MEMHOUSE_URL:-}" ] || [ -z "${MEMHOUSE_USER:-}" ]; then
|
|
|
45
52
|
fi
|
|
46
53
|
|
|
47
54
|
q() { # read-only by construction; every read path should use this
|
|
48
|
-
|
|
55
|
+
sed -E "s/([[:space:](]|^)(FROM|JOIN)[[:space:]]+(sessions|messages|tool_calls|meta|events)([[:space:];,)]|\$)/\1\2 ${MEMHOUSE_USER}_\3\4/g" \
|
|
56
|
+
| curl -sS --fail-with-body --user "$MEMHOUSE_USER:${MEMHOUSE_PASSWORD:-}" \
|
|
49
57
|
--data-binary @- "$MEMHOUSE_URL/?database=${MEMHOUSE_DB:-$MEMHOUSE_USER}&readonly=1"
|
|
50
58
|
}
|
|
51
59
|
q <<'SQL'
|
|
@@ -56,11 +64,22 @@ SQL
|
|
|
56
64
|
`readonly=1` is a ClickHouse-side setting: even a mis-generated `DROP` is refused by the
|
|
57
65
|
server, not by discipline.
|
|
58
66
|
|
|
59
|
-
**
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
67
|
+
**The `sed` is what makes every query below work in both layouts** — write room names
|
|
68
|
+
bare and it supplies the prefix when there is one. It rewrites only after `FROM`/`JOIN`,
|
|
69
|
+
so a needle like `LIKE '%messages%'` is untouched, and `FROM yigit_messages` is left
|
|
70
|
+
alone — the room name must follow whitespace, and there is none inside `yigit_messages`.
|
|
71
|
+
With no prefix set it expands to nothing and the query passes through verbatim.
|
|
72
|
+
|
|
73
|
+
It is spelled with `[[:space:]]` classes rather than `\b` because **BSD `sed` does not
|
|
74
|
+
support `\b`**: on macOS the `\b` form matched nothing and sent the query through
|
|
75
|
+
unprefixed. That fails loudly as `UNKNOWN_TABLE` rather than quietly reading wrong rows,
|
|
76
|
+
which is the right way round — but it fails.
|
|
77
|
+
|
|
78
|
+
**Reading a housemate's memory.** A share is `GRANT SELECT ON mem.<them>_*` — their
|
|
79
|
+
rooms, not the database. `SHOW TABLES` lists exactly what you may read: your own rooms plus
|
|
80
|
+
any shared with you. Name a housemate's rooms in full (`FROM yigit_messages`); the `q()`
|
|
81
|
+
rewrite leaves an already-prefixed name alone, since applying yours would point at the
|
|
82
|
+
wrong rooms. Say whose memory an answer came from.
|
|
64
83
|
|
|
65
84
|
---
|
|
66
85
|
|
|
@@ -84,13 +84,13 @@ into a transcript on disk. The CLI passes it to one process and never echoes it.
|
|
|
84
84
|
5. **Say whether the house was fresh** — `SELECT count() FROM <db>.messages` reads 0 for a
|
|
85
85
|
new one. Second line of defence behind step 3's refusal.
|
|
86
86
|
|
|
87
|
-
6. **Write the message the user will send.**
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
87
|
+
6. **Write the message the user will send.** `invite` wrote TWO files beside each other:
|
|
88
|
+
`invite-<name>.env` (the credential, one-time) and `MEMHOUSE-INVITATION.md` (the steps,
|
|
89
|
+
nothing secret — install, verify, what they own, things to try). The guide does the
|
|
90
|
+
explaining, so the message is short: what memhouse is in a sentence; that the `.env` IS a
|
|
91
|
+
password (do not forward, do not paste in chat); open `MEMHOUSE-INVITATION.md` and follow
|
|
92
|
+
it. If `memhouse --version` shows a pre-release (a `-` in it), the invitee also needs the
|
|
93
|
+
tarball this build came from (`memhouse nightly --out …`); the guide says so and names it.
|
|
94
94
|
|
|
95
95
|
If they are REPLACING an existing credential, add `memhouse ship --full` — a plain
|
|
96
96
|
incremental pass skips sessions the new house has no record of, so their history would
|
|
@@ -105,7 +105,7 @@ for this** — partial sharing has four ways to go wrong quietly, and the comman
|
|
|
105
105
|
each one:
|
|
106
106
|
|
|
107
107
|
```
|
|
108
|
-
memhouse share <user>
|
|
108
|
+
memhouse share <user> everything you own
|
|
109
109
|
memhouse share <user> --only project=memhouse just one project
|
|
110
110
|
memhouse share <user> --only session=<id> just one conversation
|
|
111
111
|
memhouse share --list who can read it, and how much
|
|
@@ -8,7 +8,7 @@ allowed-tools: Bash
|
|
|
8
8
|
|
|
9
9
|
# /mem:admin — operate the house as its administrator
|
|
10
10
|
|
|
11
|
-
A member owns one
|
|
11
|
+
A member owns a house, or rooms in one. An **administrator** owns the ClickHouse: every house, every
|
|
12
12
|
account, every grant. This skill is the second one, and it is the only skill allowed to
|
|
13
13
|
act outside the caller's own house.
|
|
14
14
|
|
|
@@ -83,12 +83,12 @@ thin. Say what could not be seen rather than guessing.
|
|
|
83
83
|
SHOW DATABASES
|
|
84
84
|
-- who you have granted (memhouse's own record of it)
|
|
85
85
|
SELECT substring(key, 7) AS user, value AS state
|
|
86
|
-
FROM
|
|
86
|
+
FROM meta FINAL WHERE key LIKE 'share:%' ORDER BY key
|
|
87
87
|
-- everyone on the server (names only — every member holds SHOW USERS)
|
|
88
88
|
SELECT name FROM system.users ORDER BY name
|
|
89
89
|
```
|
|
90
90
|
|
|
91
|
-
That `
|
|
91
|
+
That `meta` list is memhouse's own note-keeping, not ClickHouse's grant table —
|
|
92
92
|
a member cannot read `system.grants`. Present it as "what memhouse recorded", and say
|
|
93
93
|
that a grant made by hand would not appear.
|
|
94
94
|
|
package/memhouse/flags.js
CHANGED
|
@@ -16,16 +16,17 @@ const COMMAND_FLAGS = {
|
|
|
16
16
|
null: [], help: [], version: [],
|
|
17
17
|
whoami: ['admin'],
|
|
18
18
|
share: ['only', 'revoke', 'list'],
|
|
19
|
-
discover: [], stats: [], stop: [], doctor: [], 'sessions-query': [],
|
|
19
|
+
discover: [], stats: [], stop: [], doctor: [], 'sessions-query': [], rooms: [], members: ['db','admin-user','admin-password'],
|
|
20
20
|
status: [], start: [],
|
|
21
21
|
onboard: ['url', 'user', 'password', 'db', 'port', 'no-ship', 'target', 'house-port', 'tag', 'local'],
|
|
22
22
|
install: ['url', 'user', 'password', 'db', 'port', 'no-ship', 'env', 'print-sql', 'member',
|
|
23
|
-
'member-password', 'admin-user', 'admin-password', 'ensure-schema', 'rotate-password', 'force'
|
|
24
|
-
|
|
23
|
+
'member-password', 'admin-user', 'admin-password', 'ensure-schema', 'rotate-password', 'force',
|
|
24
|
+
'editors', 'claude-roots', 'channel'],
|
|
25
|
+
setup: ['url', 'user', 'password', 'db', 'port', 'admin-user', 'admin-password', 'editors', 'claude-roots', 'channel'],
|
|
25
26
|
ship: ['full', 'loop', 'ensure-schema'],
|
|
26
27
|
search: ['limit'],
|
|
27
28
|
resume: [],
|
|
28
|
-
update: ['check', 'migrate', 'no-install'],
|
|
29
|
+
update: ['check', 'migrate', 'no-install', 'channel'],
|
|
29
30
|
reset: ['all-origins'],
|
|
30
31
|
invite: ['url', 'db', 'out', 'admin-user', 'admin-password', 'adopt', 'print-sql',
|
|
31
32
|
'allow-local', 'member-password'],
|
package/memhouse/house/HOUSE.md
CHANGED
|
@@ -1,12 +1,18 @@
|
|
|
1
|
-
# The house — a database,
|
|
1
|
+
# The house — a database, and the rooms in it
|
|
2
2
|
|
|
3
3
|
**A house is a ClickHouse database.** Any database, named whatever its people name it —
|
|
4
|
-
`polat`, `team_a`, even `default`. Its rooms are
|
|
5
|
-
`
|
|
6
|
-
Merge rooms, no views, no settings profile.
|
|
4
|
+
`polat`, `team_a`, even `default`. Its rooms are tables: `sessions`, `messages`,
|
|
5
|
+
`tool_calls`. No views, no Merge rooms, no settings profile.
|
|
7
6
|
|
|
8
|
-
**
|
|
9
|
-
|
|
7
|
+
**Every member's rooms are named for them, and one grant covers them.** `mem.polat_*`
|
|
8
|
+
for polat, `mem.alice_*` for alice, in the same database — `mem` unless somebody has a
|
|
9
|
+
reason. The grant is `GRANT … ON mem.polat_* TO polat WITH GRANT OPTION`: it lets the
|
|
10
|
+
member create and rebuild their own rooms, share them, and reach nothing else. Nobody is
|
|
11
|
+
granted the database itself. A house with one member is a house of one, not a different
|
|
12
|
+
kind of house — a colleague joins as one more member and nothing about the first changes.
|
|
13
|
+
`roomNames()` in `house.js` is the only place a table name is produced.
|
|
14
|
+
|
|
15
|
+
Two columns say where every row came from:
|
|
10
16
|
|
|
11
17
|
| Column | Meaning | Comes from |
|
|
12
18
|
|---|---|---|
|
|
@@ -31,9 +37,17 @@ shipper create and evolve the rooms (`ship.js --ensure-schema`) — no admin has
|
|
|
31
37
|
memhouse code. Joining a ClickHouse that happens to run a kernel is the same three
|
|
32
38
|
statements: a database and a credential, nothing else.
|
|
33
39
|
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
40
|
+
**Housemates do not have to trust each other with the house.** They used to: everyone
|
|
41
|
+
held `ALL` on the database, and the isolation mechanism between parties who should not
|
|
42
|
+
see each other was a separate house. That shape leaked — one member could `GRANT SELECT
|
|
43
|
+
ON db.* TO <outsider>` and hand over a housemate's transcripts, with no admin and no
|
|
44
|
+
notification.
|
|
45
|
+
|
|
46
|
+
In a shared house each member is granted their own rooms and nothing else, so isolation
|
|
47
|
+
lives INSIDE one house and is checkable: `SHOW GRANTS` lists what you hold, and `SHOW
|
|
48
|
+
TABLES` does not list what you do not. A separate house is still the boundary between
|
|
49
|
+
groups that should not even know of each other — names leak across a shared house even
|
|
50
|
+
when content does not.
|
|
37
51
|
|
|
38
52
|
## One user on two machines, same session id
|
|
39
53
|
|
|
@@ -91,9 +105,9 @@ for writes and DDL.
|
|
|
91
105
|
|
|
92
106
|
Two tables that are not rooms:
|
|
93
107
|
|
|
94
|
-
- `
|
|
108
|
+
- `meta` — house-wide key/value, latest-wins: `schema_version`, and the memhouse
|
|
95
109
|
version each member last shipped with.
|
|
96
|
-
- `
|
|
110
|
+
- `events` — append-only: migrations (`pending` → `applied` | `failed`), version
|
|
97
111
|
changes, schema observations, each with actor, host and row counts. Nothing is updated
|
|
98
112
|
in place, so a migration that failed and was retried reads as exactly that.
|
|
99
113
|
|
package/memhouse/house/house.js
CHANGED
|
@@ -1,9 +1,8 @@
|
|
|
1
1
|
// The house.
|
|
2
2
|
//
|
|
3
|
-
// A house is a ClickHouse DATABASE
|
|
4
|
-
//
|
|
5
|
-
//
|
|
6
|
-
// their own credential, and two columns say who and where every row came from:
|
|
3
|
+
// A house is a ClickHouse DATABASE, `mem` by default. Its rooms are tables — `sessions`,
|
|
4
|
+
// `messages`, `tool_calls` — one set per member, named for them, and two columns say who
|
|
5
|
+
// and where every row came from:
|
|
7
6
|
//
|
|
8
7
|
// user_id String MATERIALIZED currentUser() — stamped by the server, unforgeable by
|
|
9
8
|
// clients (async_insert is pinned to 0
|
|
@@ -12,19 +11,30 @@
|
|
|
12
11
|
// host LowCardinality(String) — the machine's fingerprint, minted once
|
|
13
12
|
// per install (../host.js)
|
|
14
13
|
//
|
|
15
|
-
// That pair IS the provenance model.
|
|
16
|
-
// both write; `WHERE user_id = 'alice'` is one person, `WHERE host = '…'` is one machine.
|
|
17
|
-
// The model is collaborative — housemates trust each other with the house — and the
|
|
18
|
-
// boundary between houses is the database: joining someone's ClickHouse takes a database
|
|
19
|
-
// and a credential (`GRANT ALL ON team_a.* TO alice`), nothing else, because ALL on your
|
|
20
|
-
// own database reaches nothing outside it.
|
|
14
|
+
// That pair IS the provenance model.
|
|
21
15
|
//
|
|
22
|
-
//
|
|
23
|
-
//
|
|
24
|
-
//
|
|
25
|
-
//
|
|
26
|
-
//
|
|
27
|
-
//
|
|
16
|
+
// ONE LAYOUT. A house is a database — `mem` unless somebody has a reason — and every
|
|
17
|
+
// member's rooms in it carry the member's name: `mem.alice_messages`, `mem.polat_messages`.
|
|
18
|
+
// A member holds one grant, on the wildcard `mem.<name>_*`, and nothing else in the
|
|
19
|
+
// database. That is the whole access model, and roomNames() below is the only place a
|
|
20
|
+
// table name is produced.
|
|
21
|
+
//
|
|
22
|
+
// alone on a laptop mem.polat_* a house of one
|
|
23
|
+
// a team on one server mem.polat_*, mem.alice_*
|
|
24
|
+
// an agency on a kernel the same, in the kernel's `mem`
|
|
25
|
+
//
|
|
26
|
+
// Nobody ever holds ALL ON <db>.*. That grant is dynamic — it covers rooms created later —
|
|
27
|
+
// and it is what turned two members in one database into a leak: either could read and
|
|
28
|
+
// re-grant the other's transcripts, measured. The wildcard reaches nothing outside the
|
|
29
|
+
// member's own name: measured, a member cannot create, read, list or re-grant a
|
|
30
|
+
// housemate's rooms, and can read their own grant back with SHOW GRANTS in one line.
|
|
31
|
+
//
|
|
32
|
+
// THIS FILE HAS SAID THREE OTHER THINGS. Shared tables with row policies (0.4.0: policies
|
|
33
|
+
// are permissive and OR'd, a catch-all fails open). Suffixed per-member rooms with Merge
|
|
34
|
+
// rooms over them (0.8.0: removed because clients had to resolve names first and
|
|
35
|
+
// isolation "solved a problem the product did not have" — the second half was wrong).
|
|
36
|
+
// Then a house per member with `ALL ON db.*`, plus a prefixed variant beside it, plus the
|
|
37
|
+
// machinery to keep the two from meeting. All three were special cases of this one.
|
|
28
38
|
//
|
|
29
39
|
// `sessions_v` IS A SAVED QUERY, NOT AN OBJECT — substituted into the same `FROM … AS c`
|
|
30
40
|
// position a view name would occupy, running under the caller's own credential. It was
|
|
@@ -38,7 +48,11 @@ const ROOM_TYPES = ['sessions', 'messages', 'tool_calls'];
|
|
|
38
48
|
* them, the shipper's guards do not require them, and a member who cannot create them
|
|
39
49
|
* still ships normally.
|
|
40
50
|
*/
|
|
41
|
-
|
|
51
|
+
// `<member>_meta` / `<member>_events` since the one-layout: every row in them is about ONE
|
|
52
|
+
// member (their schema version, their machines, their last ship, their shares), so the old
|
|
53
|
+
// `house_` prefix named the wrong unit. A pre-one-layout house still has `house_meta` /
|
|
54
|
+
// `house_events`; the shipper's legacy guard names the rename.
|
|
55
|
+
const META_TYPES = ['meta', 'events'];
|
|
42
56
|
|
|
43
57
|
/**
|
|
44
58
|
* What generation of the schema a house is at. Bumped only when existing rooms have to be
|
|
@@ -75,7 +89,7 @@ const SUPPORTED_SCHEMAS = [2];
|
|
|
75
89
|
|
|
76
90
|
/**
|
|
77
91
|
* The floor a house may set under its writers, recorded by `memhouse migrate` in
|
|
78
|
-
*
|
|
92
|
+
* meta['min_writer_schema']. Today it equals SCHEMA_VERSION; a future
|
|
79
93
|
* back-compatible generation can hold it one step lower for a grace window.
|
|
80
94
|
*/
|
|
81
95
|
const MIN_WRITER_SCHEMA = 2;
|
|
@@ -233,15 +247,19 @@ const ROOM_MATERIALIZED = {
|
|
|
233
247
|
tool_calls: ['user_id'],
|
|
234
248
|
};
|
|
235
249
|
|
|
236
|
-
function currentParse(table, epochSource = table) {
|
|
250
|
+
function currentParse(table, epochSource = table, physical = null) {
|
|
251
|
+
// `table` names the ROOM (for the materialized-column lookup); `physical` is what goes
|
|
252
|
+
// into the FROM, which differs whenever a prefix is in play.
|
|
237
253
|
const extra = (ROOM_MATERIALIZED[table] || ['user_id']).join(', ');
|
|
254
|
+
const from = physical || table;
|
|
255
|
+
const epochFrom = physical ? physical.replace(new RegExp(`${table}$`), epochSource) : epochSource;
|
|
238
256
|
return `(
|
|
239
257
|
SELECT *, ${extra}
|
|
240
|
-
FROM ${
|
|
258
|
+
FROM ${from} FINAL
|
|
241
259
|
WHERE origin != 'ship'
|
|
242
260
|
OR (session_id, user_id, epoch) IN (
|
|
243
261
|
SELECT session_id, user_id, max(epoch)
|
|
244
|
-
FROM ${
|
|
262
|
+
FROM ${epochFrom}
|
|
245
263
|
WHERE origin = 'ship'
|
|
246
264
|
GROUP BY session_id, user_id)
|
|
247
265
|
)`;
|
|
@@ -302,24 +320,54 @@ const READ_SETTINGS = { final: 1, join_use_nulls: 1 };
|
|
|
302
320
|
* name it produces. See ship.js's delete, which must BIND the user rather than call
|
|
303
321
|
* currentUser() inside a mutation, where it is not evaluated in the caller's context.
|
|
304
322
|
*/
|
|
323
|
+
/**
|
|
324
|
+
* The table a room type lands in. THE naming rule, and the only copy of it.
|
|
325
|
+
*
|
|
326
|
+
* ONE LAYOUT. Every member's rooms carry the member's own name: `mem.alice_messages`,
|
|
327
|
+
* and `mem.polat_messages` even when polat is the only one in there. A house with one
|
|
328
|
+
* member is a house of one, not a different kind of house — so there is no second case
|
|
329
|
+
* here, no prefix to configure, and nothing for an invite file to carry.
|
|
330
|
+
*
|
|
331
|
+
* Depends on nothing but the member, so a caller holding only a config can name a table
|
|
332
|
+
* without a server round-trip. That matters: when the rule lived only inside roomNames(),
|
|
333
|
+
* callers that could not afford `SELECT currentUser()` spelled tables by hand instead —
|
|
334
|
+
* eight of them, all swallowing the error — and a house silently had no metadata plane.
|
|
335
|
+
*/
|
|
336
|
+
function physicalRoom(type, user) {
|
|
337
|
+
if (!user) throw new Error(`physicalRoom(${type}): a room belongs to a member, and none was given`);
|
|
338
|
+
return `${user}_${type}`;
|
|
339
|
+
}
|
|
340
|
+
|
|
341
|
+
/** The wildcard every grant to a member is scoped to: `<member>_*`. */
|
|
342
|
+
function roomPattern(user) {
|
|
343
|
+
if (!user) throw new Error('roomPattern: a member is required');
|
|
344
|
+
return `${user}_*`;
|
|
345
|
+
}
|
|
346
|
+
|
|
305
347
|
function roomNames(user) {
|
|
306
|
-
|
|
348
|
+
if (!user) throw new Error('roomNames: a member is required');
|
|
349
|
+
const out = { member: user, user, pattern: roomPattern(user) };
|
|
350
|
+
// Every room name in the codebase comes from physicalRoom() above — through this
|
|
351
|
+
// function where a caller has the rooms, directly where it has only a member name.
|
|
352
|
+
const phys = (t) => physicalRoom(t, user);
|
|
353
|
+
out.physical = {};
|
|
354
|
+
for (const t of [...ROOM_TYPES, ...META_TYPES]) out.physical[t] = phys(t);
|
|
307
355
|
// The raw table names — for INSERT, for DDL, and for the shipper's own bookkeeping
|
|
308
356
|
// reads, which have to see every epoch to decide which one to write next.
|
|
309
|
-
for (const t of ROOM_TYPES) out[`${t}_raw`] = t;
|
|
357
|
+
for (const t of ROOM_TYPES) out[`${t}_raw`] = phys(t);
|
|
310
358
|
// What everything else gets. `sessions` is unfiltered: it is one metadata row per
|
|
311
359
|
// session by construction, latest-wins, and carries no epoch anyone may read.
|
|
312
|
-
out.sessions = 'sessions';
|
|
313
|
-
out.messages = currentParse('messages');
|
|
360
|
+
out.sessions = phys('sessions');
|
|
361
|
+
out.messages = currentParse('messages', 'messages', phys('messages'));
|
|
314
362
|
// tool_calls takes its epoch from MESSAGES, not from itself. A parse that produces
|
|
315
363
|
// messages but NO tool calls is ordinary — a compaction can remove every assistant turn
|
|
316
364
|
// that called something — and it writes zero rows into this room at the new epoch. Asked
|
|
317
365
|
// for its own max(epoch), the room would answer with the SUPERSEDED epoch and serve the
|
|
318
366
|
// old parse's tool calls beside the new parse's messages. Both rooms are written by the
|
|
319
367
|
// same pass at the same epoch, so messages is the authority for both.
|
|
320
|
-
out.tool_calls = currentParse('tool_calls', 'messages');
|
|
368
|
+
out.tool_calls = currentParse('tool_calls', 'messages', phys('tool_calls'));
|
|
321
369
|
// The house's own record of itself — plain names, nothing to filter.
|
|
322
|
-
for (const t of META_TYPES) { out[t] = t; out[`${t}_raw`] = t; }
|
|
370
|
+
for (const t of META_TYPES) { out[t] = phys(t); out[`${t}_raw`] = phys(t); }
|
|
323
371
|
out.sessions_v = sessionsRollup(out);
|
|
324
372
|
return out;
|
|
325
373
|
}
|
|
@@ -337,7 +385,42 @@ async function resolveRooms(client) {
|
|
|
337
385
|
return roomNames(await currentUser(client));
|
|
338
386
|
}
|
|
339
387
|
|
|
388
|
+
/**
|
|
389
|
+
* The text-index DDL for servers before the tokenizer grammar changed. 25.8 spells a
|
|
390
|
+
* tokenizer as a quoted name with options — `text(tokenizer = 'ngram', ngram_size = 3)`,
|
|
391
|
+
* `text(tokenizer = 'default')` — and rejects the function form with "Expected literal";
|
|
392
|
+
* 26.x spells it as a function — `ngrams(3)`, `splitByNonAlpha` — and rejects the quoted
|
|
393
|
+
* names as "Unknown tokenizer". Neither parses the other. The template carries the
|
|
394
|
+
* current grammar; this rewrites a statement to the older one when a server refuses it.
|
|
395
|
+
* `default` in the old grammar is the non-alphanumeric splitter, which is what
|
|
396
|
+
* `splitByNonAlpha` names in the new one, so the indexes built are the same.
|
|
397
|
+
*/
|
|
398
|
+
function legacyTextIndexDialect(sql) {
|
|
399
|
+
return sql
|
|
400
|
+
.replace(/TYPE text\(tokenizer = ngrams\((\d+)\)\)/g, "TYPE text(tokenizer = 'ngram', ngram_size = $1)")
|
|
401
|
+
.replace(/TYPE text\(tokenizer = splitByNonAlpha\)/g, "TYPE text(tokenizer = 'default')");
|
|
402
|
+
}
|
|
403
|
+
|
|
404
|
+
/**
|
|
405
|
+
* Which text-index grammar a server speaks, from `SELECT version()`. Measured on real
|
|
406
|
+
* builds: 25.8 and 25.9 want the quoted-name form, 25.10 onwards the function form, and
|
|
407
|
+
* each rejects the other. Anything unparseable is treated as current; the shipper still
|
|
408
|
+
* falls back on a grammar refusal, so a wrong guess costs one round-trip, not the install.
|
|
409
|
+
*/
|
|
410
|
+
function textIndexDialectFor(version) {
|
|
411
|
+
const m = /^(\d+)\.(\d+)/.exec(String(version || '').trim());
|
|
412
|
+
if (!m) return 'modern';
|
|
413
|
+
const major = Number(m[1]); const minor = Number(m[2]);
|
|
414
|
+
return (major < 25 || (major === 25 && minor < 10)) ? 'legacy' : 'modern';
|
|
415
|
+
}
|
|
416
|
+
|
|
417
|
+
/** True when a server's refusal is the grammar, not a privilege or a real mistake. */
|
|
418
|
+
function isTextIndexGrammarRefusal(message) {
|
|
419
|
+
return /Expected literal|supports only 'default'|Unknown tokenizer/.test(String(message || ''));
|
|
420
|
+
}
|
|
421
|
+
|
|
340
422
|
module.exports = {
|
|
423
|
+
physicalRoom, roomPattern, legacyTextIndexDialect, isTextIndexGrammarRefusal, textIndexDialectFor,
|
|
341
424
|
ROOM_TYPES, META_TYPES, SCHEMA_VERSION, SUPPORTED_SCHEMAS, MIN_WRITER_SCHEMA,
|
|
342
425
|
MIGRATIONS, ROOM_KEYS, keyProblem,
|
|
343
426
|
READ_SETTINGS, MEMBER_PIN,
|
|
@@ -39,8 +39,8 @@
|
|
|
39
39
|
// the old table until the rename; a late-writes pass copies anything newer than the
|
|
40
40
|
// snapshot, and ReplacingMergeTree makes that idempotent.
|
|
41
41
|
//
|
|
42
|
-
// EVERYTHING IS RECORDED. One
|
|
43
|
-
// after, per object; schema_version in
|
|
42
|
+
// EVERYTHING IS RECORDED. One events pending row before work, applied/failed
|
|
43
|
+
// after, per object; schema_version in meta only after a migration completes.
|
|
44
44
|
|
|
45
45
|
const fs = require('fs');
|
|
46
46
|
const path = require('path');
|
|
@@ -59,7 +59,7 @@ const { ROOM_TYPES, keyProblem, createStatement } = require('./house');
|
|
|
59
59
|
* toVersion the SCHEMA_VERSION a house is at once this has run
|
|
60
60
|
* detect(q, ctx) -> what still needs doing (any truthy non-empty value), or
|
|
61
61
|
* null/[] when the house is already past this migration.
|
|
62
|
-
* THE ROOMS ARE THE TRUTH, never
|
|
62
|
+
* THE ROOMS ARE THE TRUTH, never meta: a hand-migrated
|
|
63
63
|
* house has no record, and a restored backup can carry a
|
|
64
64
|
* record newer than its tables.
|
|
65
65
|
* plan(found) -> lines of human text for the confirm prompt
|
|
@@ -81,7 +81,7 @@ function listMigrations() {
|
|
|
81
81
|
const seen = new Set();
|
|
82
82
|
let lastVersion = 0;
|
|
83
83
|
for (const m of out) {
|
|
84
|
-
// The id is spliced into SQL (
|
|
84
|
+
// The id is spliced into SQL (events predicates) and into kept-table suffixes.
|
|
85
85
|
// It is repo-controlled, but "repo-controlled" is one compromised dependency away
|
|
86
86
|
// from "attacker-controlled" — validate like any other identifier.
|
|
87
87
|
if (!/^[a-z0-9][a-z0-9-]*$/.test(m.id)) throw new Error(`migration id '${m.id}' must match [a-z0-9-]+`);
|
|
@@ -116,7 +116,7 @@ async function detectPending(q, ctx, { component = null } = {}) {
|
|
|
116
116
|
* Refuse when another actor's run is unfinished. An `applied` newer than the last
|
|
117
117
|
* `pending` means done; a dangling `pending` means someone is (or was) mid-copy, and two
|
|
118
118
|
* concurrent rebuilds of one room end with one of them renaming the other's work.
|
|
119
|
-
* Best-effort: a house without
|
|
119
|
+
* Best-effort: a house without events cannot answer, and that is not a reason to
|
|
120
120
|
* block the migration that will create it.
|
|
121
121
|
*/
|
|
122
122
|
async function unfinishedBy(q, ctx, migrationId) {
|
|
@@ -124,7 +124,8 @@ async function unfinishedBy(q, ctx, migrationId) {
|
|
|
124
124
|
const rows = await q.rows(
|
|
125
125
|
`SELECT argMax(status, event_at) AS status, argMax(actor, event_at) AS actor,
|
|
126
126
|
formatDateTime(max(event_at), '%Y-%m-%d %H:%i') AS at
|
|
127
|
-
FROM
|
|
127
|
+
FROM ${(ctx.rooms && ctx.rooms.physical && ctx.rooms.physical.events) || 'events'}
|
|
128
|
+
WHERE kind = 'migration' AND id = '${migrationId}'`);
|
|
128
129
|
const r = rows[0];
|
|
129
130
|
if (r && r.status === 'pending' && r.actor && r.actor !== ctx.member) return r;
|
|
130
131
|
} catch { /* no events table yet */ }
|
|
@@ -12,8 +12,8 @@
|
|
|
12
12
|
// not insertable even with insert_allow_materialized_columns.
|
|
13
13
|
const DERIVED = ['text_ngram', 'text_word'];
|
|
14
14
|
|
|
15
|
-
//
|
|
16
|
-
// Everything else in
|
|
15
|
+
// meta keys that are DURABLE facts of the house — worth carrying to the new host.
|
|
16
|
+
// Everything else in meta is a per-host heartbeat (client_version, client_schema,
|
|
17
17
|
// last_ship:<writer>) that the new shipper rewrites for itself on its first run; carrying
|
|
18
18
|
// a stale one would misreport which machine last shipped until that run lands.
|
|
19
19
|
function isDurableMetaKey(key) {
|
|
@@ -154,10 +154,10 @@ ORDER BY (session_id, user_id, origin, epoch, idx);
|
|
|
154
154
|
-- authoritative and always current, and a mirror of them would be a second truth that
|
|
155
155
|
-- goes stale. What is recorded is what those cannot say: intent, sequence, outcome.
|
|
156
156
|
|
|
157
|
-
CREATE TABLE IF NOT EXISTS
|
|
157
|
+
CREATE TABLE IF NOT EXISTS meta
|
|
158
158
|
(
|
|
159
159
|
-- House-wide keys only ('schema_version', 'house_created_at'). Anything per-member
|
|
160
|
-
-- or per-machine belongs in
|
|
160
|
+
-- or per-machine belongs in events, which is append-only — this room is
|
|
161
161
|
-- latest-wins, so two members writing one key would overwrite each other.
|
|
162
162
|
key String,
|
|
163
163
|
value String,
|
|
@@ -168,7 +168,7 @@ CREATE TABLE IF NOT EXISTS house_meta
|
|
|
168
168
|
ENGINE = ReplacingMergeTree(updated_at)
|
|
169
169
|
ORDER BY (key);
|
|
170
170
|
|
|
171
|
-
CREATE TABLE IF NOT EXISTS
|
|
171
|
+
CREATE TABLE IF NOT EXISTS events
|
|
172
172
|
(
|
|
173
173
|
event_at DateTime64(3, 'UTC') DEFAULT now64(3),
|
|
174
174
|
-- 'migration' — a room rebuild, one row per transition (pending → applied|failed)
|