memhouse 0.7.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +29 -11
- package/bin/memhouse.js +129 -108
- package/memhouse/DESIGN.md +29 -30
- package/memhouse/delivery/PROMPT.md +7 -9
- package/memhouse/delivery/kernel-install.md +14 -29
- package/memhouse/delivery/plugin/skills/search/SKILL.md +25 -38
- package/memhouse/delivery/plugin/skills/sessions/SKILL.md +15 -28
- package/memhouse/delivery/plugin/skills/sql/SKILL.md +23 -36
- package/memhouse/host.js +1 -1
- package/memhouse/house/HOUSE.md +74 -0
- package/memhouse/house/house.js +165 -0
- package/memhouse/{per-member/schema-member.sql.tpl → house/schema.sql.tpl} +8 -11
- package/memhouse/resume.js +1 -1
- package/memhouse/server/queries.js +8 -2
- package/memhouse/shipper/ship.js +24 -30
- package/package.json +1 -1
- package/memhouse/per-member/INSTALL.md +0 -228
- package/memhouse/per-member/PLAN.md +0 -174
- package/memhouse/per-member/PROVISIONING.md +0 -177
- package/memhouse/per-member/README.md +0 -48
- package/memhouse/per-member/SCHEMA.md +0 -120
- package/memhouse/per-member/SHARING.md +0 -102
- package/memhouse/per-member/provision.js +0 -265
- package/memhouse/per-member/rooms.js +0 -235
- package/memhouse/per-member/schema-merge.sql.tpl +0 -24
package/README.md
CHANGED
|
@@ -36,6 +36,13 @@ makes it worse — [EACCES on install](#eacces-on-install) tells them apart.
|
|
|
36
36
|
memhouse deploy --local # a loopback-bound ClickHouse, then install and ship
|
|
37
37
|
```
|
|
38
38
|
|
|
39
|
+
The house it stands up is **yours by name**: its superuser defaults to your OS
|
|
40
|
+
username (`polat`, not `memhouse_root` — override with `--user`), and the house
|
|
41
|
+
database defaults to the same name, so your memory lives at `polat.messages` and
|
|
42
|
+
`WHERE user_id = 'polat'` reads like it should. The name
|
|
43
|
+
and password are fixed when the data volume is first initialised, like any ClickHouse
|
|
44
|
+
container, and reused on every redeploy after that.
|
|
45
|
+
|
|
39
46
|
Otherwise point at one you already run, or hand the SQL to whoever administers it:
|
|
40
47
|
|
|
41
48
|
```bash
|
|
@@ -157,19 +164,30 @@ immediate.
|
|
|
157
164
|
|
|
158
165
|
## How your memory is stored
|
|
159
166
|
|
|
160
|
-
**
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
167
|
+
**A house is a database; its rooms are three shared tables.** `sessions`, `messages`
|
|
168
|
+
and `tool_calls` live in whatever database you point at — your own name by default
|
|
169
|
+
(`polat.messages`), a team's (`team_a.messages`), even `default`. Everyone in the
|
|
170
|
+
house writes into the same tables with their own credential, and every row says where
|
|
171
|
+
it came from: `user_id`, stamped by the server (`MATERIALIZED currentUser()`, with
|
|
172
|
+
async inserts pinned off so the stamp cannot be skipped), and `host`, the machine's
|
|
173
|
+
install fingerprint.
|
|
174
|
+
|
|
175
|
+
`WHERE user_id = 'alice'` is one person. `WHERE host = '…'` is one machine. No filter
|
|
176
|
+
is the whole house — which is exactly what a team dashboard wants.
|
|
164
177
|
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
178
|
+
Sharing is not a feature bolted on top; it IS the house. A team makes a database,
|
|
179
|
+
grants each person `ALL` on it, and their shippers all write into the same rooms:
|
|
180
|
+
|
|
181
|
+
```sql
|
|
182
|
+
CREATE DATABASE team_a;
|
|
183
|
+
CREATE USER alice IDENTIFIED BY '…';
|
|
184
|
+
GRANT ALL ON team_a.* TO alice; -- repeat per housemate
|
|
185
|
+
```
|
|
169
186
|
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
187
|
+
`ALL` on your own house reaches nothing outside it — the database is the boundary,
|
|
188
|
+
which is also why joining a ClickHouse someone else runs (a kernel's, a team's) needs
|
|
189
|
+
no negotiation beyond a database and a credential. Housemates are collaborators;
|
|
190
|
+
groups that should not see each other get separate houses.
|
|
173
191
|
|
|
174
192
|
### Imported history is protected
|
|
175
193
|
|
package/bin/memhouse.js
CHANGED
|
@@ -23,7 +23,7 @@ const SHIP_JS = path.join(REPO_ROOT, 'memhouse', 'shipper', 'ship.js');
|
|
|
23
23
|
const SERVER_JS = path.join(REPO_ROOT, 'memhouse', 'server', 'server.js');
|
|
24
24
|
const DELIVERY = path.join(REPO_ROOT, 'memhouse', 'delivery');
|
|
25
25
|
const PKG = require(path.join(REPO_ROOT, 'package.json'));
|
|
26
|
-
const { roomNames, ROOM_TYPES,
|
|
26
|
+
const { roomNames, ROOM_TYPES, MEMBER_PIN, installCommand } = require(path.join(REPO_ROOT, 'memhouse', 'house', 'house'));
|
|
27
27
|
const envfile = require(path.join(REPO_ROOT, 'memhouse', 'envfile'));
|
|
28
28
|
|
|
29
29
|
const HOME_DIR = process.env.MEMHOUSE_HOME || path.join(os.homedir(), '.memhouse');
|
|
@@ -65,7 +65,10 @@ function resolveConfig() {
|
|
|
65
65
|
url: pick('url', 'MEMHOUSE_URL', 'http://localhost:8123'),
|
|
66
66
|
user: pick('user', 'MEMHOUSE_USER', 'memhouse_root'),
|
|
67
67
|
password: pick('password', 'MEMHOUSE_PASSWORD', ''),
|
|
68
|
-
|
|
68
|
+
// The house defaults to the USER'S OWN NAME (`polat` ships into `polat.messages`),
|
|
69
|
+
// because a house is a database and the natural first house is your own. Any name
|
|
70
|
+
// works — user 'alice' with house 'default' is a supported pairing.
|
|
71
|
+
db: pick('db', 'MEMHOUSE_DB', pick('user', 'MEMHOUSE_USER', 'memhouse_root')),
|
|
69
72
|
port: pick('port', 'MEMHOUSE_PORT', '4640'),
|
|
70
73
|
// Did anything actually SAY which house this is, or are the values above just the
|
|
71
74
|
// defaults? The defaults are not neutral — localhost:8123 as memhouse_root is a real
|
|
@@ -600,38 +603,20 @@ function generatePassword() {
|
|
|
600
603
|
* same templates provision.js applies, so the two cannot drift into different houses.
|
|
601
604
|
*/
|
|
602
605
|
function memberSql(db, member, password) {
|
|
603
|
-
|
|
604
|
-
//
|
|
605
|
-
//
|
|
606
|
-
// the play UI, curl — so every name is qualified here and there is no `USE`.
|
|
607
|
-
//
|
|
608
|
-
// `currentDatabase()` in the Merge engine is the trap, and it is silent:
|
|
609
|
-
// it is evaluated at CREATE time, not at read time, so a block run without the house
|
|
610
|
-
// selected builds Merge rooms pointing at `default` that return zero rows forever. The
|
|
611
|
-
// literal name goes in instead.
|
|
606
|
+
// The template is written unqualified because the shipper applies it with the house
|
|
607
|
+
// already selected. A human pastes this somewhere unknown — clickhouse-client, the
|
|
608
|
+
// play UI, curl — so every name is qualified here and there is no `USE`.
|
|
612
609
|
const qualify = (sql) => sql
|
|
613
|
-
.replace(/CREATE TABLE IF NOT EXISTS (\w+)/g, `CREATE TABLE IF NOT EXISTS ${db}.$1`)
|
|
614
|
-
|
|
615
|
-
.replace(/Merge\(currentDatabase\(\)/g, `Merge('${db}'`);
|
|
616
|
-
const rooms = qualify(fs.readFileSync(path.join(here, 'schema-member.sql.tpl'), 'utf-8')
|
|
617
|
-
.replaceAll('{{MEMBER}}', member));
|
|
618
|
-
const merge = qualify(fs.readFileSync(path.join(here, 'schema-merge.sql.tpl'), 'utf-8')
|
|
619
|
-
.replaceAll('{{TEMPLATE_MEMBER}}', member));
|
|
620
|
-
// The SAME grant set provision.js issues. These two paths build the same house, and when
|
|
621
|
-
// only one of them was narrowed, `install --print-sql` — the path INSTALL.md calls the
|
|
622
|
-
// common case — still emitted GRANT ALL, which includes CREATE TABLE on the member's own
|
|
623
|
-
// room name. A member provisioned that way could drop their room, recreate it as
|
|
624
|
-
// Merge('<db>','^messages_'), and double every other member's rows in the team room:
|
|
625
|
-
// exactly the attack the narrowing was for, fully reachable, one release later.
|
|
626
|
-
const grants = ROOM_TYPES.flatMap((t) => [
|
|
627
|
-
`GRANT ${MEMBER_PRIVS} ON ${db}.${t}_${member} TO ${member};`,
|
|
628
|
-
`GRANT SELECT ON ${db}.${t}_${member} TO ${member} WITH GRANT OPTION;`,
|
|
629
|
-
]).join('\n');
|
|
630
|
-
const mergeGrants = Object.values(mergeRooms())
|
|
631
|
-
.map((n) => `GRANT SELECT ON ${db}.${n} TO ${member};`).join('\n');
|
|
610
|
+
.replace(/CREATE TABLE IF NOT EXISTS (\w+)/g, `CREATE TABLE IF NOT EXISTS ${db}.$1`);
|
|
611
|
+
const rooms = qualify(fs.readFileSync(path.join(REPO_ROOT, 'memhouse', 'house', 'schema.sql.tpl'), 'utf-8'));
|
|
632
612
|
return `-- memhouse: everything '${member}' needs in house '${db}'. Run as a user with
|
|
633
613
|
-- ACCESS MANAGEMENT (a stock 'default' with access_management=1 will do).
|
|
634
614
|
--
|
|
615
|
+
-- A house is a database; its rooms are three shared tables. Everyone granted on the
|
|
616
|
+
-- database writes into the same tables, and every row carries who (user_id, stamped by
|
|
617
|
+
-- the server) and where from (host, the install's fingerprint). Adding a housemate later
|
|
618
|
+
-- is the same two statements: CREATE USER, GRANT ALL ON the house.
|
|
619
|
+
--
|
|
635
620
|
-- Every name is qualified and there is no USE, so this runs anywhere: clickhouse-client,
|
|
636
621
|
-- the play UI, curl, a GUI. Order matters only in that the database comes first.
|
|
637
622
|
--
|
|
@@ -655,26 +640,16 @@ CREATE USER IF NOT EXISTS ${member} IDENTIFIED BY '${password.replace(/'/g, "\\'
|
|
|
655
640
|
|
|
656
641
|
${rooms.trim()}
|
|
657
642
|
|
|
658
|
-
--
|
|
659
|
-
--
|
|
660
|
-
|
|
661
|
-
|
|
662
|
-
|
|
663
|
-
|
|
664
|
-
--
|
|
665
|
-
--
|
|
666
|
-
|
|
667
|
-
|
|
668
|
-
${mergeGrants}
|
|
669
|
-
|
|
670
|
-
-- Resource ceilings. Confidentiality isolation is a grant that is absent; this is the
|
|
671
|
-
-- availability half, and it did not exist until 0.4.7 — a member could raise
|
|
672
|
-
-- max_memory_usage, zero max_execution_time and take a shared server down for everyone.
|
|
673
|
-
-- CONST, not MAX: a plain default is advisory, and MAX is not enough either because 0 means
|
|
674
|
-
-- UNLIMITED in ClickHouse and 0 satisfies any MAX. Measured — with a MAX ceiling in force,
|
|
675
|
-
-- SETTINGS max_memory_usage = 0 was accepted and the ceiling was gone.
|
|
676
|
-
CREATE SETTINGS PROFILE OR REPLACE ${MEMBER_PROFILE} SETTINGS ${MEMBER_PROFILE_SETTINGS};
|
|
677
|
-
ALTER USER ${member} SETTINGS PROFILE '${MEMBER_PROFILE}';
|
|
643
|
+
-- The whole database. ALL on your own house reaches nothing outside it — the database is
|
|
644
|
+
-- the boundary — and it is what lets the shipper create and evolve its own tables.
|
|
645
|
+
GRANT ALL ON ${db}.* TO ${member};
|
|
646
|
+
|
|
647
|
+
-- Pin async_insert on the user (ADD SETTING merges; a bare SETTINGS clause would replace
|
|
648
|
+
-- the user's whole list). The pin keeps the user_id stamp honest: a MATERIALIZED
|
|
649
|
+
-- currentUser() is computed during the INSERT, and an async flush stores it as the empty
|
|
650
|
+
-- string. CONST refuses the override; a client that never mentions the setting is held
|
|
651
|
+
-- at 0 and its insert lands stamped.
|
|
652
|
+
ALTER USER ${member} ADD SETTING ${MEMBER_PIN};
|
|
678
653
|
`;
|
|
679
654
|
}
|
|
680
655
|
|
|
@@ -702,22 +677,19 @@ async function adminBootstrap(cfg, admin) {
|
|
|
702
677
|
catch { console.log(bad(`house '${cfg.db}' does not exist and '${admin.user}' cannot create it: ${e.message}`)); return null; }
|
|
703
678
|
}
|
|
704
679
|
|
|
705
|
-
// 2. the user.
|
|
706
|
-
//
|
|
707
|
-
//
|
|
708
|
-
//
|
|
709
|
-
// SELECT ON system.users. That is the isolation working — but the raw 497 names neither
|
|
710
|
-
// side, while INSTALL.md promises install "refuses, names both sides, and creates
|
|
711
|
-
// nothing". Say who you are not, and what the owner has to run instead.
|
|
680
|
+
// 2. the user. A shared house makes an existing user unremarkable — rows are told apart
|
|
681
|
+
// by the server-stamped user_id, so "the handle is taken" hands nobody anything. What
|
|
682
|
+
// still matters is proof: connecting AS the member below is what validates whichever
|
|
683
|
+
// password is in play, existing user or fresh one.
|
|
712
684
|
let exists;
|
|
713
685
|
try {
|
|
714
686
|
exists = (await chRows(adminCfg, `SELECT name FROM system.users WHERE name = '${admin.member}'`, { database: '' })).length > 0;
|
|
715
687
|
} catch (e) {
|
|
716
688
|
if (/Not enough privileges|ACCESS_DENIED|system\.users/i.test(e.message || '')) {
|
|
717
689
|
console.log(bad(`'${admin.user}' is not an admin of this house — it cannot read system.users, so it cannot provision a member`));
|
|
718
|
-
console.log('
|
|
719
|
-
console.log(`
|
|
720
|
-
console.log(' If you already HAVE a
|
|
690
|
+
console.log(' Joining a house is two statements for whoever owns it:');
|
|
691
|
+
console.log(` CREATE USER ${admin.member} IDENTIFIED BY '…'; GRANT ALL ON ${cfg.db}.* TO ${admin.member};`);
|
|
692
|
+
console.log(' If you already HAVE a credential on this house, you do not need admin:');
|
|
721
693
|
console.log(` memhouse install --url ${adminCfg.url} --user ${admin.member} --password …`);
|
|
722
694
|
return null;
|
|
723
695
|
}
|
|
@@ -725,19 +697,14 @@ async function adminBootstrap(cfg, admin) {
|
|
|
725
697
|
}
|
|
726
698
|
let password = flags['member-password'] || null;
|
|
727
699
|
if (exists) {
|
|
728
|
-
if (
|
|
729
|
-
console.log(bad(`ClickHouse user '${admin.member}' already exists
|
|
730
|
-
console.log(
|
|
731
|
-
console.log(' currentUser(), so their rows and yours would be indistinguishable.');
|
|
732
|
-
console.log(' If this is you on a new machine, prove it:');
|
|
733
|
-
console.log(` memhouse install --adopt-user --member ${admin.member} --member-password '…' …`);
|
|
734
|
-
console.log(` If it is someone else, pick another handle: --member <name>`);
|
|
700
|
+
if (!password) {
|
|
701
|
+
console.log(bad(`ClickHouse user '${admin.member}' already exists — pass --member-password to install as them.`));
|
|
702
|
+
console.log(` Or pick another handle: --member <name>`);
|
|
735
703
|
return null;
|
|
736
704
|
}
|
|
737
|
-
if (!password) { console.log(bad('--adopt-user requires --member-password — the password is the proof')); return null; }
|
|
738
705
|
try { await ch({ ...cfg, user: admin.member, password }, 'SELECT 1', { database: '' }); }
|
|
739
|
-
catch { console.log(bad(
|
|
740
|
-
console.log(ok(`
|
|
706
|
+
catch { console.log(bad(`'${admin.member}' did not authenticate with that password`)); return null; }
|
|
707
|
+
console.log(ok(`using the existing ClickHouse user '${admin.member}' (password verified)`));
|
|
741
708
|
} else {
|
|
742
709
|
if (!password) {
|
|
743
710
|
password = generatePassword();
|
|
@@ -753,24 +720,18 @@ async function adminBootstrap(cfg, admin) {
|
|
|
753
720
|
createdUser = admin.member;
|
|
754
721
|
}
|
|
755
722
|
|
|
756
|
-
// 3.
|
|
757
|
-
//
|
|
758
|
-
|
|
759
|
-
|
|
760
|
-
|
|
761
|
-
|
|
762
|
-
|
|
763
|
-
|
|
764
|
-
|
|
765
|
-
// and it runs before this. Observed: provision.js died on a missing dependency, this
|
|
766
|
-
// printed the all-clear, and the next install refused with the identity-takeover
|
|
767
|
-
// message because the user it said it had not created was there. Say what exists, and
|
|
768
|
-
// name the two ways forward.
|
|
769
|
-
console.log(bad('provisioning failed'));
|
|
723
|
+
// 3. the grant and the pin. GRANT ALL on the house is the whole tenancy model: it
|
|
724
|
+
// reaches nothing outside the database, and it is what lets the member's own shipper
|
|
725
|
+
// create and evolve the rooms (--ensure-schema below).
|
|
726
|
+
try {
|
|
727
|
+
await q(`GRANT ALL ON ${cfg.db}.* TO ${admin.member}`, { database: '' });
|
|
728
|
+
await q(`ALTER USER ${admin.member} ADD SETTING ${MEMBER_PIN}`, { database: '' });
|
|
729
|
+
console.log(ok(`granted the house: ALL ON ${cfg.db}.* to '${admin.member}', async_insert pinned`));
|
|
730
|
+
} catch (e) {
|
|
731
|
+
console.log(bad(`could not grant the house to '${admin.member}': ${e.message}`));
|
|
770
732
|
if (createdUser) {
|
|
771
|
-
console.log(` ClickHouse user '${createdUser}' WAS created before this failed
|
|
772
|
-
console.log(
|
|
773
|
-
console.log(` memhouse install --adopt-user --member ${createdUser} --member-password '<the password above>' …`);
|
|
733
|
+
console.log(` ClickHouse user '${createdUser}' WAS created before this failed.`);
|
|
734
|
+
console.log(` Continue once fixed: memhouse install --member ${createdUser} --member-password '<the password above>' …`);
|
|
774
735
|
console.log(` Or undo it as the admin: DROP USER ${createdUser}`);
|
|
775
736
|
} else {
|
|
776
737
|
console.log(' nothing was written');
|
|
@@ -779,11 +740,15 @@ async function adminBootstrap(cfg, admin) {
|
|
|
779
740
|
}
|
|
780
741
|
|
|
781
742
|
// The step that makes this trustworthy: stop being admin, and prove the credential we
|
|
782
|
-
// are about to persist
|
|
743
|
+
// are about to persist can build and reach the rooms itself.
|
|
783
744
|
const memberCfg = { ...cfg, user: admin.member, password };
|
|
745
|
+
if (run(SHIP_JS, ['--ensure-schema'], memberCfg) !== 0) {
|
|
746
|
+
console.log(bad(`'${admin.member}' could not create the rooms in '${cfg.db}' — nothing written to disk`));
|
|
747
|
+
return null;
|
|
748
|
+
}
|
|
784
749
|
try {
|
|
785
|
-
const seen = await chRows(memberCfg, `SELECT count() AS n FROM system.tables WHERE database = '${cfg.db}' AND name IN ('
|
|
786
|
-
if (Number(seen[0]?.n) !== 3) { console.log(bad(`'${admin.member}' cannot see
|
|
750
|
+
const seen = await chRows(memberCfg, `SELECT count() AS n FROM system.tables WHERE database = '${cfg.db}' AND name IN ('sessions','messages','tool_calls')`, { database: '' });
|
|
751
|
+
if (Number(seen[0]?.n) !== 3) { console.log(bad(`'${admin.member}' cannot see the three rooms — nothing written`)); return null; }
|
|
787
752
|
} catch (e) { console.log(bad(`'${admin.member}' could not connect after provisioning: ${e.message}`)); return null; }
|
|
788
753
|
console.log(ok(`verified as '${admin.member}' — admin credential discarded, not stored`));
|
|
789
754
|
return memberCfg;
|
|
@@ -841,6 +806,16 @@ async function cmdInstall({ interactive }) {
|
|
|
841
806
|
let cfg = resolveConfig();
|
|
842
807
|
const adminUser = flags['admin-user'];
|
|
843
808
|
|
|
809
|
+
// The house name is spliced unquoted into CREATE DATABASE and GRANT ALL ON <db>.* —
|
|
810
|
+
// validate it HERE, on every install mode, not only where deploy happens to choose it.
|
|
811
|
+
// Found by the acceptance run, driven end to end: `install --db system` sailed through,
|
|
812
|
+
// granted a member CHECK/SHOW/SELECT/INSERT/ALTER/DROP/TRUNCATE/… ON system.* and
|
|
813
|
+
// created sessions/messages/tool_calls INSIDE the server's system database, exit 0.
|
|
814
|
+
// On a shared or kernel ClickHouse that is a granted-everything-on-system user minted
|
|
815
|
+
// by a memhouse one-liner.
|
|
816
|
+
try { require(path.join(REPO_ROOT, 'memhouse', 'house', 'house')).assertUsableName(cfg.db, 'house'); }
|
|
817
|
+
catch (e) { console.log(bad(e.message)); return 1; }
|
|
818
|
+
|
|
844
819
|
// Option 1 — print the SQL and stop. For the common case: you have admin on this
|
|
845
820
|
// ClickHouse and would rather run four statements yourself than hand a credential to an
|
|
846
821
|
// installer. Nothing is written and nothing is contacted.
|
|
@@ -879,7 +854,7 @@ async function cmdInstall({ interactive }) {
|
|
|
879
854
|
if (keyProblem) {
|
|
880
855
|
console.log(bad(`the rooms are here, but their sorting keys are wrong: ${keyProblem}`));
|
|
881
856
|
console.log(' a ship pass would corrupt them, so `memhouse ship` will refuse.');
|
|
882
|
-
console.log(` rebuild them from ${path.join(REPO_ROOT, 'memhouse', '
|
|
857
|
+
console.log(` rebuild them from ${path.join(REPO_ROOT, 'memhouse', 'house', 'schema.sql.tpl')}, then: memhouse ship --full`);
|
|
883
858
|
console.log(' no config was written — nothing here reads as installed.');
|
|
884
859
|
return 1;
|
|
885
860
|
}
|
|
@@ -893,8 +868,8 @@ async function cmdInstall({ interactive }) {
|
|
|
893
868
|
}
|
|
894
869
|
if (flags['no-ship'] !== true && run(SHIP_JS, [], cfg) !== 0) return 1;
|
|
895
870
|
console.log(ok('installed — next: memhouse start (dashboard + shipper loop)'));
|
|
896
|
-
console.log('
|
|
897
|
-
console.log(`
|
|
871
|
+
console.log(' adding a housemate later is two statements for the admin:');
|
|
872
|
+
console.log(` CREATE USER <name> IDENTIFIED BY '…'; GRANT ALL ON ${cfg.db}.* TO <name>;`);
|
|
898
873
|
return 0;
|
|
899
874
|
}
|
|
900
875
|
|
|
@@ -903,7 +878,7 @@ async function cmdInstall({ interactive }) {
|
|
|
903
878
|
// authenticating as memhouse_root instead: the three flags did nothing, --yes satisfied
|
|
904
879
|
// haveAll so no prompt asked who you were, and resolveConfig's default was used AS A
|
|
905
880
|
// CREDENTIAL. Refuse rather than do something else silently.
|
|
906
|
-
const adminOnly = ['
|
|
881
|
+
const adminOnly = ['member', 'member-password'].filter((k) => flags[k] !== undefined);
|
|
907
882
|
if (!adminUser && adminOnly.length) {
|
|
908
883
|
console.log(bad(`--${adminOnly.join(', --')} ${adminOnly.length > 1 ? 'are' : 'is'} only read with --admin-user`));
|
|
909
884
|
console.log(' Those flags provision a member, which needs house admin. Without them this');
|
|
@@ -1016,7 +991,7 @@ async function cmdInstall({ interactive }) {
|
|
|
1016
991
|
if (keyProblem) {
|
|
1017
992
|
console.log(bad(`the rooms are here, but their sorting keys are wrong: ${keyProblem}`));
|
|
1018
993
|
console.log(' a ship pass would corrupt them, so `memhouse ship` will refuse.');
|
|
1019
|
-
console.log(` rebuild them from ${path.join(REPO_ROOT, 'memhouse', '
|
|
994
|
+
console.log(` rebuild them from ${path.join(REPO_ROOT, 'memhouse', 'house', 'schema.sql.tpl')}, then: memhouse ship --full`);
|
|
1020
995
|
console.log(' no config was written — nothing here reads as installed.');
|
|
1021
996
|
return 1;
|
|
1022
997
|
}
|
|
@@ -1306,7 +1281,7 @@ async function cmdDoctor() {
|
|
|
1306
1281
|
add(t === objects.length, t === objects.length
|
|
1307
1282
|
? `schema: 3/3 rooms in '${cfg.db}' (${objects.join(', ')})`
|
|
1308
1283
|
: `schema: ${t}/${objects.length} rooms in '${cfg.db}' — missing ${absent.join(', ')}`,
|
|
1309
|
-
`run: memhouse install
|
|
1284
|
+
`run: memhouse install — with ALL on the house, ship --ensure-schema builds them`);
|
|
1310
1285
|
} catch (e) { add(false, 'schema check', `${e.message} — run: memhouse install`); }
|
|
1311
1286
|
// Columns, not just rooms. A room with a column missing accepts every insert and
|
|
1312
1287
|
// discards that field — measured, 85 rows shipped with the value thrown away while
|
|
@@ -1314,7 +1289,7 @@ async function cmdDoctor() {
|
|
|
1314
1289
|
// counted rooms.
|
|
1315
1290
|
try {
|
|
1316
1291
|
const { templateColumns } = require(path.join(REPO_ROOT, 'memhouse', 'shipper', 'ship.js'));
|
|
1317
|
-
const tpl = fs.readFileSync(path.join(REPO_ROOT, 'memhouse', '
|
|
1292
|
+
const tpl = fs.readFileSync(path.join(REPO_ROOT, 'memhouse', 'house', 'schema.sql.tpl'), 'utf-8');
|
|
1318
1293
|
const want = templateColumns(tpl);
|
|
1319
1294
|
// Names AND types AND the MATERIALIZED kind. Comparing names alone left every other
|
|
1320
1295
|
// kind of drift invisible, with measured consequences: a `UInt64` column narrowed to
|
|
@@ -1379,7 +1354,7 @@ async function cmdDoctor() {
|
|
|
1379
1354
|
add(false, `sorting keys: only ${checked}/${ROOM_TYPES.length} rooms are MergeTree — the rest have no sorting key at all`,
|
|
1380
1355
|
'a room was replaced by a Merge/View/Log engine; rebuild it from the schema template as the house owner');
|
|
1381
1356
|
} else add(wrongKeys.length === 0, `sorting keys${wrongKeys.length ? `: wrong on ${wrongKeys.join(', ')}` : ` carry origin correctly (${checked}/${ROOM_TYPES.length} rooms)`}`,
|
|
1382
|
-
`rebuild those rooms from the schema template, then: memhouse ship --full\n template: ${path.join(REPO_ROOT, 'memhouse', '
|
|
1357
|
+
`rebuild those rooms from the schema template, then: memhouse ship --full\n template: ${path.join(REPO_ROOT, 'memhouse', 'house', 'schema.sql.tpl')}`);
|
|
1383
1358
|
} catch (e) { add(false, 'sorting keys', e.message); }
|
|
1384
1359
|
try {
|
|
1385
1360
|
// countIf, not any(). `any()` returns an arbitrary row's value, so on a house with
|
|
@@ -1917,8 +1892,8 @@ async function cmdReset() {
|
|
|
1917
1892
|
} catch { /* pre-origin house: nothing to protect, ensureSchema will add the column */ }
|
|
1918
1893
|
|
|
1919
1894
|
if (flags.yes !== true) {
|
|
1920
|
-
// Name the rooms
|
|
1921
|
-
//
|
|
1895
|
+
// Name the rooms, and the scope honestly: the tables are shared, but reset only ever
|
|
1896
|
+
// deletes rows whose user_id is the CALLER's — a housemate's rows are untouched.
|
|
1922
1897
|
const scope = allOrigins
|
|
1923
1898
|
? `EVERY row, including ${imported} imported one(s) that re-shipping CANNOT restore,`
|
|
1924
1899
|
: 'the rows the shipper wrote';
|
|
@@ -2257,10 +2232,43 @@ async function cmdUninstall() {
|
|
|
2257
2232
|
: process.env.MEMHOUSE_PASSWORD ? 'MEMHOUSE_PASSWORD' : null;
|
|
2258
2233
|
if (initialised && rotateAsk) {
|
|
2259
2234
|
console.log(bad(`${rotateAsk} cannot change the credential of an existing house — the image only applies it when it initialises the data directory.`));
|
|
2260
|
-
console.log(
|
|
2235
|
+
console.log(` to rotate: ALTER USER ${priorCfg.user || 'memhouse_root'} IDENTIFIED BY '…' inside the house, then: memhouse setup --password …`);
|
|
2261
2236
|
console.log(' to start over (DESTROYS the memory): memhouse deploy --down');
|
|
2262
2237
|
process.exitCode = 2; break;
|
|
2263
2238
|
}
|
|
2239
|
+
// Who owns this house. A local deploy is the pilot's own machine and their own
|
|
2240
|
+
// memory, so the house user is THEM — suggested from the OS username (`polat`, not
|
|
2241
|
+
// `memhouse_root`) — created by the container itself at first init with full
|
|
2242
|
+
// rights. No admin/member split, no provisioning dance: a URI and a credential is
|
|
2243
|
+
// the whole install. The name is fixed at volume init exactly like the password,
|
|
2244
|
+
// so an existing house keeps the user it was born with.
|
|
2245
|
+
const osName = (os.userInfo().username || '').replace(/[^A-Za-z0-9_]/g, '_').replace(/^[0-9_]+/, '');
|
|
2246
|
+
let houseUser;
|
|
2247
|
+
if (initialised) {
|
|
2248
|
+
// The env FILE, not resolveConfig(): flags outrank the file there, so comparing
|
|
2249
|
+
// `--user other` against priorCfg.user compares the flag against itself and the
|
|
2250
|
+
// refusal below can never fire — while the volume stays initialised with the
|
|
2251
|
+
// original name and every later connection fails auth.
|
|
2252
|
+
const storedUser = readEnvFile().MEMHOUSE_USER;
|
|
2253
|
+
// No stored user means the env file is gone or emptied — the stored PASSWORD is
|
|
2254
|
+
// gone with it, so the `initialised && !reusable` refusal below already owns this
|
|
2255
|
+
// case. Refusing `--user` here on a guessed 'memhouse_root' would tell the pilot
|
|
2256
|
+
// their house "was initialised with memhouse_root" on no evidence at all.
|
|
2257
|
+
if (storedUser && flags.user && flags.user !== true && flags.user !== storedUser) {
|
|
2258
|
+
console.log(bad(`--user cannot rename the user of an existing house — the image creates it only when it initialises the data directory.`));
|
|
2259
|
+
console.log(` this house was initialised with '${storedUser}'. To start over (DESTROYS the memory): memhouse deploy --down`);
|
|
2260
|
+
process.exitCode = 2; break;
|
|
2261
|
+
}
|
|
2262
|
+
houseUser = storedUser || 'memhouse_root';
|
|
2263
|
+
} else if (flags.user && flags.user !== true) {
|
|
2264
|
+
houseUser = String(flags.user);
|
|
2265
|
+
} else if (flags.yes !== true && process.stdin.isTTY) {
|
|
2266
|
+
houseUser = (await ask(`House user (this becomes your member name)`, osName || 'memhouse_root')).trim() || osName || 'memhouse_root';
|
|
2267
|
+
} else {
|
|
2268
|
+
houseUser = osName || 'memhouse_root';
|
|
2269
|
+
}
|
|
2270
|
+
try { require(path.join(REPO_ROOT, 'memhouse', 'house', 'house')).assertUsableName(houseUser, 'house user'); }
|
|
2271
|
+
catch (e) { console.log(bad(`'${houseUser}' cannot be the house user — ${e.message}`)); process.exitCode = 2; break; }
|
|
2264
2272
|
// An initialised volume with no credential to reuse — env file deleted, emptied, or
|
|
2265
2273
|
// never written — is the same lockout by another route: a generated password would
|
|
2266
2274
|
// be ignored by the server and then written over the config as if it worked.
|
|
@@ -2287,7 +2295,17 @@ async function cmdUninstall() {
|
|
|
2287
2295
|
const tag = flags.tag || process.env.MEMHOUSE_CH_TAG || dep.managedTag() || dep.DEFAULT_TAG;
|
|
2288
2296
|
// The database is part of the destination, not a detail of it — see the service
|
|
2289
2297
|
// check below, which must compare it.
|
|
2290
|
-
|
|
2298
|
+
// STATED sources only, then the house user. priorCfg.db is unusable here: resolveConfig
|
|
2299
|
+
// computes a DEFAULT for it (the user's name, itself defaulted), so going through it
|
|
2300
|
+
// handed a fresh deploy `memhouse_root` as the house name while the superuser was
|
|
2301
|
+
// 'polat' — the memory would live in a database named after a role account nobody
|
|
2302
|
+
// chose.
|
|
2303
|
+
const targetDb = flags.db || process.env.MEMHOUSE_DB || readEnvFile().MEMHOUSE_DB || houseUser;
|
|
2304
|
+
// Same check install runs, but EARLY — deploy initialises the container volume and
|
|
2305
|
+
// writes the env file before its install step, so a refusal that waits for install
|
|
2306
|
+
// arrives after the damage.
|
|
2307
|
+
try { require(path.join(REPO_ROOT, 'memhouse', 'house', 'house')).assertUsableName(targetDb, 'house'); }
|
|
2308
|
+
catch (e) { console.log(bad(e.message)); process.exitCode = 2; break; }
|
|
2291
2309
|
{
|
|
2292
2310
|
// A service-managed shipper keeps the environment it was installed with, so ANY
|
|
2293
2311
|
// switch that repoints the config leaves it shipping somewhere else — not only an
|
|
@@ -2341,7 +2359,7 @@ async function cmdUninstall() {
|
|
|
2341
2359
|
}
|
|
2342
2360
|
const pw = reusable || crypto.randomBytes(16).toString('hex');
|
|
2343
2361
|
if (reusable) console.log(ok('reusing the existing house credential (its data volume is already initialised)'));
|
|
2344
|
-
const r = dep.up({ password: pw, port, tag });
|
|
2362
|
+
const r = dep.up({ password: pw, port, tag, user: houseUser });
|
|
2345
2363
|
if (!r.ok) { console.log(bad(r.msg)); process.exitCode = 1; break; }
|
|
2346
2364
|
console.log(ok(`ClickHouse starting via ${r.engine} on ${r.url} (loopback only)`));
|
|
2347
2365
|
|
|
@@ -2353,7 +2371,10 @@ async function cmdUninstall() {
|
|
|
2353
2371
|
// house that eventually came up is unreachable forever. The image applies the
|
|
2354
2372
|
// password only at first init, so there is no way back from that.
|
|
2355
2373
|
if (!reusable) {
|
|
2356
|
-
|
|
2374
|
+
// db INCLUDED: this file is read back by the install below, and writing
|
|
2375
|
+
// priorCfg's computed default here made that install refuse over a mismatch
|
|
2376
|
+
// with a file this same command had just written.
|
|
2377
|
+
writeEnvFile({ ...priorCfg, url: r.url, user: houseUser, password: pw, db: targetDb });
|
|
2357
2378
|
console.log(ok(`credential saved to ${ENV_FILE} before waiting — the volume is initialised with it`));
|
|
2358
2379
|
}
|
|
2359
2380
|
|
|
@@ -2403,10 +2424,10 @@ async function cmdUninstall() {
|
|
|
2403
2424
|
// anticipates it) and it is NOT the narrow set INSTALL.md's grant-set argument
|
|
2404
2425
|
// describes. A user who later adds a second member should know which house they
|
|
2405
2426
|
// have.
|
|
2406
|
-
console.log(warn(`this house is yours alone: '${
|
|
2407
|
-
console.log(' Adding a
|
|
2408
|
-
console.log(`
|
|
2409
|
-
flags.url = r.url; flags.user =
|
|
2427
|
+
console.log(warn(`this house is yours alone: '${houseUser}' is its superuser, and that is the credential being saved`));
|
|
2428
|
+
console.log(' Adding a housemate later is two statements:');
|
|
2429
|
+
console.log(` CREATE USER <name> IDENTIFIED BY '…'; GRANT ALL ON ${targetDb}.* TO <name>;`);
|
|
2430
|
+
flags.url = r.url; flags.user = houseUser; flags.password = pw;
|
|
2410
2431
|
flags.db = targetDb;
|
|
2411
2432
|
flags.yes = true;
|
|
2412
2433
|
process.exitCode = await cmdInstall({ interactive: false });
|
package/memhouse/DESIGN.md
CHANGED
|
@@ -5,10 +5,12 @@ on your machines — across **all 17 editors** agentlytics supports — parsed l
|
|
|
5
5
|
shipped to a typed ClickHouse store, shareable with a team, installable on the
|
|
6
6
|
ultimagent kernel as an **agency**, and visible through the agentlytics dashboard.
|
|
7
7
|
|
|
8
|
-
**
|
|
9
|
-
`
|
|
10
|
-
|
|
11
|
-
|
|
8
|
+
**A house is a database; its rooms are three shared tables** — `sessions`, `messages`,
|
|
9
|
+
`tool_calls`. Everyone in the house writes into the same tables with their own
|
|
10
|
+
credential; the server-stamped `user_id` says who, the install fingerprint `host` says
|
|
11
|
+
which machine. Sharing IS the house; the boundary between groups is the database. (The
|
|
12
|
+
layout has moved twice: shared-tables-with-row-policies fell in 0.4.0, per-member
|
|
13
|
+
suffixed rooms fell in 0.8.0 — see `house/HOUSE.md` for why.)
|
|
12
14
|
|
|
13
15
|
**Agency, precisely.** In constellation terms (`../TERMINOLOGY.md`) memhouse is a
|
|
14
16
|
**house** — the `mem` database — **plus a resident**: the shipper. That pairing
|
|
@@ -24,23 +26,21 @@ on = "loop" # a daemon — not an insert trigger, not a schedule
|
|
|
24
26
|
```
|
|
25
27
|
|
|
26
28
|
The session rollup is a **routine** — house machinery, not a second resident. Since
|
|
27
|
-
0.4.0 it is not even an object: it is SQL text (`
|
|
28
|
-
|
|
29
|
-
nothing. memhouse has **no materialized views at all** — in fact no stored views at all
|
|
29
|
+
0.4.0 it is not even an object: it is SQL text (`house/house.js`), computed during your
|
|
30
|
+
query, for your query, writing nothing. memhouse has **no materialized views at all** — in fact no stored views at all
|
|
30
31
|
— so the shipper is not merely a resident, it is the only candidate in the tree. Strip
|
|
31
32
|
it and every remaining moving part is a routine over rows nobody is writing any more.
|
|
32
33
|
|
|
33
34
|
Note that `user_id MATERIALIZED currentUser()` is a materialized *column*: it computes
|
|
34
|
-
during your own insert and is part of the table's definition. It is
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
property, not what makes this an agency.
|
|
35
|
+
during your own insert and is part of the table's definition. It is what tells housemates'
|
|
36
|
+
rows apart in the shared tables — provenance, stamped by the server, with async_insert
|
|
37
|
+
pinned so it cannot be skipped. An identity property, not what makes this an agency.
|
|
38
38
|
|
|
39
39
|
**Positioning:** an alternative agency **competing with memory-house**. It borrows
|
|
40
40
|
memory-house's proven ideas (server-stamped identity, idempotent shipping,
|
|
41
41
|
skills/plugin delivery) and agentlytics' proven assets (adapters, UI, cost engine) —
|
|
42
|
-
but **not** its
|
|
43
|
-
one
|
|
42
|
+
but **not** its row-policy machinery: memhouse's tables are shared by people who chose
|
|
43
|
+
one house, and the boundary between houses is the database itself. If it wins, it can become memory-house v4; until then the
|
|
44
44
|
two run side by side as separate agencies, each in its own house. Naming: the product is
|
|
45
45
|
**memhouse**; the agency is **`memhouse`** and its house is the **`mem`** database
|
|
46
46
|
(identifiers can't carry a dash; the owner kept the longer name, `memhouse_root`).
|
|
@@ -60,20 +60,19 @@ them free and further along; **bet 3 is the one nobody else attempts.**
|
|
|
60
60
|
write axis this bet is precisely *moving work from routine to resident*:
|
|
61
61
|
memory-house parses when you query, memhouse parses before anyone asks and
|
|
62
62
|
writes the result down.
|
|
63
|
-
2. **Typed common schema** (`
|
|
64
|
-
|
|
65
|
-
memory-house
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
3. **Kernel-installable agency.**
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
themselves (grant-option) or the owner.
|
|
63
|
+
2. **Typed common schema** (`house/schema.sql.tpl` — three shared tables per house).
|
|
64
|
+
Physical typed columns (what memory-house derives in views) + `tool_calls`
|
|
65
|
+
(memory-house has no tool table) + one `extra JSON` escape hatch per room so
|
|
66
|
+
unnormalized adapter fields are never lost. `ReplacingMergeTree(ingested_at)`;
|
|
67
|
+
`messages` keyed `(session_id, user_id, origin, seq)` so a re-ship of a
|
|
68
|
+
grown/changed session replaces stale rows (latest-wins) and two housemates' rows
|
|
69
|
+
never collapse into one. FTS text indexes built in (CH ≥ 26.2).
|
|
70
|
+
3. **Kernel-installable agency.** Joining any ClickHouse — a kernel's included — is a
|
|
71
|
+
database plus a credential: `CREATE USER`, `GRANT ALL ON <house>.* `, done. The
|
|
72
|
+
shipper is the resident that lands with it and creates its own rooms. Identity is
|
|
73
|
+
`user_id MATERIALIZED currentUser()` with `async_insert = 0 CONST` pinned on the
|
|
74
|
+
user; machines are told apart by the `host` fingerprint. The boundary between
|
|
75
|
+
agencies on one server is the database — a house's ALL reaches nothing outside it.
|
|
77
76
|
4. **Borrowed UI, zero fork.** The React SPA consumes REST JSON, not tables. The
|
|
78
77
|
memhouse server implements the **same `/api/*` contract** as agentlytics'
|
|
79
78
|
`server.js` (same routes, same response shapes) over the memhouse schema, and
|
|
@@ -83,9 +82,9 @@ them free and further along; **bet 3 is the one nobody else attempts.**
|
|
|
83
82
|
|
|
84
83
|
| Path | What | Notes |
|
|
85
84
|
|---|---|---|
|
|
86
|
-
| `
|
|
87
|
-
| `
|
|
88
|
-
| `
|
|
85
|
+
| `house/house.js` | room names + the session rollup + the user pin | one rule for both transports |
|
|
86
|
+
| `house/schema.sql.tpl` | the house's three shared tables (typed) | applied by each member's own shipper |
|
|
87
|
+
| `house/HOUSE.md` | the model, and what it replaced | |
|
|
89
88
|
| `shipper/ship.js` | parse-on-client shipper CLI — the resident (`worker`) | reuses `../../editors`; incremental; idempotent |
|
|
90
89
|
| `server/server.js` | REST API + dashboard | same API contract as agentlytics; serves `../../public` |
|
|
91
90
|
| `delivery/` | delivery kit | installer, skills, plugin, AGENT-INSTALL.md, prompt |
|
|
@@ -26,22 +26,20 @@ FIRST. Only say you don't know after a search comes back empty.
|
|
|
26
26
|
(full-text over messages), `/memhouse:sessions` (list/filter sessions),
|
|
27
27
|
`/memhouse:sql` (free-form read-only SQL). Without skills, query directly.
|
|
28
28
|
|
|
29
|
-
**Table names
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
`sessions_v` object** — the rollup is a saved query
|
|
29
|
+
**Table names.** The house's rooms are three plain, shared tables — `messages`,
|
|
30
|
+
`sessions`, `tool_calls` — in the connection's database. Everyone in the house writes
|
|
31
|
+
into the same tables; `user_id` (server-stamped) says whose row it is and `host` says
|
|
32
|
+
which machine. Query the whole house by default; add `WHERE user_id = '<name>'` for one
|
|
33
|
+
person. **There is no `sessions_v` object** — the rollup is a saved query, and
|
|
34
34
|
`memhouse sessions-query` prints it ready to drop into a `FROM (...) AS c`.
|
|
35
|
-
`all_messages` / `all_sessions` / `all_tool_calls` read across every member you hold a
|
|
36
|
-
grant for.
|
|
37
35
|
|
|
38
36
|
- Find sessions about a topic (FTS, lowercase your terms):
|
|
39
|
-
`SELECT DISTINCT session_id, any(project), min(ts) FROM
|
|
37
|
+
`SELECT DISTINCT session_id, any(project), min(ts) FROM messages
|
|
40
38
|
WHERE hasToken(text_word, 'clickhouse') GROUP BY session_id
|
|
41
39
|
ORDER BY 3 DESC LIMIT 10 SETTINGS final=1, join_use_nulls=1 FORMAT PrettyCompact`
|
|
42
40
|
- Recent sessions: `SELECT session_id, source, project, started, first_prompt
|
|
43
41
|
FROM $(memhouse sessions-query) AS c ORDER BY started DESC LIMIT 20 SETTINGS final=1, join_use_nulls=1`
|
|
44
|
-
- Replay one session: `SELECT role, text FROM
|
|
42
|
+
- Replay one session: `SELECT role, text FROM messages
|
|
45
43
|
WHERE session_id = '<id>' ORDER BY seq SETTINGS final=1, join_use_nulls=1`
|
|
46
44
|
|
|
47
45
|
**Rules.**
|