memhouse 0.7.1 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +29 -11
- package/bin/memhouse.js +129 -122
- package/memhouse/DESIGN.md +29 -30
- package/memhouse/delivery/PROMPT.md +7 -9
- package/memhouse/delivery/kernel-install.md +14 -29
- package/memhouse/delivery/plugin/skills/search/SKILL.md +25 -38
- package/memhouse/delivery/plugin/skills/sessions/SKILL.md +15 -28
- package/memhouse/delivery/plugin/skills/sql/SKILL.md +23 -36
- package/memhouse/host.js +1 -1
- package/memhouse/house/HOUSE.md +74 -0
- package/memhouse/house/house.js +165 -0
- package/memhouse/{per-member/schema-member.sql.tpl → house/schema.sql.tpl} +8 -11
- package/memhouse/resume.js +1 -1
- package/memhouse/server/queries.js +8 -2
- package/memhouse/shipper/ship.js +24 -30
- package/package.json +1 -1
- package/memhouse/per-member/INSTALL.md +0 -228
- package/memhouse/per-member/PLAN.md +0 -174
- package/memhouse/per-member/PROVISIONING.md +0 -177
- package/memhouse/per-member/README.md +0 -48
- package/memhouse/per-member/SCHEMA.md +0 -120
- package/memhouse/per-member/SHARING.md +0 -149
- package/memhouse/per-member/provision.js +0 -297
- package/memhouse/per-member/rooms.js +0 -252
- package/memhouse/per-member/schema-merge.sql.tpl +0 -24
package/README.md
CHANGED
|
@@ -36,6 +36,13 @@ makes it worse — [EACCES on install](#eacces-on-install) tells them apart.
|
|
|
36
36
|
memhouse deploy --local # a loopback-bound ClickHouse, then install and ship
|
|
37
37
|
```
|
|
38
38
|
|
|
39
|
+
The house it stands up is **yours by name**: its superuser defaults to your OS
|
|
40
|
+
username (`polat`, not `memhouse_root` — override with `--user`), and the house
|
|
41
|
+
database defaults to the same name, so your memory lives at `polat.messages` and
|
|
42
|
+
`WHERE user_id = 'polat'` reads like it should. The name
|
|
43
|
+
and password are fixed when the data volume is first initialised, like any ClickHouse
|
|
44
|
+
container, and reused on every redeploy after that.
|
|
45
|
+
|
|
39
46
|
Otherwise point at one you already run, or hand the SQL to whoever administers it:
|
|
40
47
|
|
|
41
48
|
```bash
|
|
@@ -157,19 +164,30 @@ immediate.
|
|
|
157
164
|
|
|
158
165
|
## How your memory is stored
|
|
159
166
|
|
|
160
|
-
**
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
167
|
+
**A house is a database; its rooms are three shared tables.** `sessions`, `messages`
|
|
168
|
+
and `tool_calls` live in whatever database you point at — your own name by default
|
|
169
|
+
(`polat.messages`), a team's (`team_a.messages`), even `default`. Everyone in the
|
|
170
|
+
house writes into the same tables with their own credential, and every row says where
|
|
171
|
+
it came from: `user_id`, stamped by the server (`MATERIALIZED currentUser()`, with
|
|
172
|
+
async inserts pinned off so the stamp cannot be skipped), and `host`, the machine's
|
|
173
|
+
install fingerprint.
|
|
174
|
+
|
|
175
|
+
`WHERE user_id = 'alice'` is one person. `WHERE host = '…'` is one machine. No filter
|
|
176
|
+
is the whole house — which is exactly what a team dashboard wants.
|
|
164
177
|
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
178
|
+
Sharing is not a feature bolted on top; it IS the house. A team makes a database,
|
|
179
|
+
grants each person `ALL` on it, and their shippers all write into the same rooms:
|
|
180
|
+
|
|
181
|
+
```sql
|
|
182
|
+
CREATE DATABASE team_a;
|
|
183
|
+
CREATE USER alice IDENTIFIED BY '…';
|
|
184
|
+
GRANT ALL ON team_a.* TO alice; -- repeat per housemate
|
|
185
|
+
```
|
|
169
186
|
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
187
|
+
`ALL` on your own house reaches nothing outside it — the database is the boundary,
|
|
188
|
+
which is also why joining a ClickHouse someone else runs (a kernel's, a team's) needs
|
|
189
|
+
no negotiation beyond a database and a credential. Housemates are collaborators;
|
|
190
|
+
groups that should not see each other get separate houses.
|
|
173
191
|
|
|
174
192
|
### Imported history is protected
|
|
175
193
|
|
package/bin/memhouse.js
CHANGED
|
@@ -23,7 +23,7 @@ const SHIP_JS = path.join(REPO_ROOT, 'memhouse', 'shipper', 'ship.js');
|
|
|
23
23
|
const SERVER_JS = path.join(REPO_ROOT, 'memhouse', 'server', 'server.js');
|
|
24
24
|
const DELIVERY = path.join(REPO_ROOT, 'memhouse', 'delivery');
|
|
25
25
|
const PKG = require(path.join(REPO_ROOT, 'package.json'));
|
|
26
|
-
const { roomNames, ROOM_TYPES,
|
|
26
|
+
const { roomNames, ROOM_TYPES, MEMBER_PIN, installCommand } = require(path.join(REPO_ROOT, 'memhouse', 'house', 'house'));
|
|
27
27
|
const envfile = require(path.join(REPO_ROOT, 'memhouse', 'envfile'));
|
|
28
28
|
|
|
29
29
|
const HOME_DIR = process.env.MEMHOUSE_HOME || path.join(os.homedir(), '.memhouse');
|
|
@@ -65,7 +65,10 @@ function resolveConfig() {
|
|
|
65
65
|
url: pick('url', 'MEMHOUSE_URL', 'http://localhost:8123'),
|
|
66
66
|
user: pick('user', 'MEMHOUSE_USER', 'memhouse_root'),
|
|
67
67
|
password: pick('password', 'MEMHOUSE_PASSWORD', ''),
|
|
68
|
-
|
|
68
|
+
// The house defaults to the USER'S OWN NAME (`polat` ships into `polat.messages`),
|
|
69
|
+
// because a house is a database and the natural first house is your own. Any name
|
|
70
|
+
// works — user 'alice' with house 'default' is a supported pairing.
|
|
71
|
+
db: pick('db', 'MEMHOUSE_DB', pick('user', 'MEMHOUSE_USER', 'memhouse_root')),
|
|
69
72
|
port: pick('port', 'MEMHOUSE_PORT', '4640'),
|
|
70
73
|
// Did anything actually SAY which house this is, or are the values above just the
|
|
71
74
|
// defaults? The defaults are not neutral — localhost:8123 as memhouse_root is a real
|
|
@@ -600,38 +603,20 @@ function generatePassword() {
|
|
|
600
603
|
* same templates provision.js applies, so the two cannot drift into different houses.
|
|
601
604
|
*/
|
|
602
605
|
function memberSql(db, member, password) {
|
|
603
|
-
|
|
604
|
-
//
|
|
605
|
-
//
|
|
606
|
-
// the play UI, curl — so every name is qualified here and there is no `USE`.
|
|
607
|
-
//
|
|
608
|
-
// `currentDatabase()` in the Merge engine is the trap, and it is silent:
|
|
609
|
-
// it is evaluated at CREATE time, not at read time, so a block run without the house
|
|
610
|
-
// selected builds Merge rooms pointing at `default` that return zero rows forever. The
|
|
611
|
-
// literal name goes in instead.
|
|
606
|
+
// The template is written unqualified because the shipper applies it with the house
|
|
607
|
+
// already selected. A human pastes this somewhere unknown — clickhouse-client, the
|
|
608
|
+
// play UI, curl — so every name is qualified here and there is no `USE`.
|
|
612
609
|
const qualify = (sql) => sql
|
|
613
|
-
.replace(/CREATE TABLE IF NOT EXISTS (\w+)/g, `CREATE TABLE IF NOT EXISTS ${db}.$1`)
|
|
614
|
-
|
|
615
|
-
.replace(/Merge\(currentDatabase\(\)/g, `Merge('${db}'`);
|
|
616
|
-
const rooms = qualify(fs.readFileSync(path.join(here, 'schema-member.sql.tpl'), 'utf-8')
|
|
617
|
-
.replaceAll('{{MEMBER}}', member));
|
|
618
|
-
const merge = qualify(fs.readFileSync(path.join(here, 'schema-merge.sql.tpl'), 'utf-8')
|
|
619
|
-
.replaceAll('{{TEMPLATE_MEMBER}}', member));
|
|
620
|
-
// The SAME grant set provision.js issues. These two paths build the same house, and when
|
|
621
|
-
// only one of them was narrowed, `install --print-sql` — the path INSTALL.md calls the
|
|
622
|
-
// common case — still emitted GRANT ALL, which includes CREATE TABLE on the member's own
|
|
623
|
-
// room name. A member provisioned that way could drop their room, recreate it as
|
|
624
|
-
// Merge('<db>','^messages_'), and double every other member's rows in the team room:
|
|
625
|
-
// exactly the attack the narrowing was for, fully reachable, one release later.
|
|
626
|
-
const grants = ROOM_TYPES.flatMap((t) => [
|
|
627
|
-
`GRANT ${MEMBER_PRIVS} ON ${db}.${t}_${member} TO ${member};`,
|
|
628
|
-
`GRANT SELECT ON ${db}.${t}_${member} TO ${member} WITH GRANT OPTION;`,
|
|
629
|
-
]).join('\n');
|
|
630
|
-
const mergeGrants = Object.values(mergeRooms())
|
|
631
|
-
.map((n) => `GRANT SELECT ON ${db}.${n} TO ${member};`).join('\n');
|
|
610
|
+
.replace(/CREATE TABLE IF NOT EXISTS (\w+)/g, `CREATE TABLE IF NOT EXISTS ${db}.$1`);
|
|
611
|
+
const rooms = qualify(fs.readFileSync(path.join(REPO_ROOT, 'memhouse', 'house', 'schema.sql.tpl'), 'utf-8'));
|
|
632
612
|
return `-- memhouse: everything '${member}' needs in house '${db}'. Run as a user with
|
|
633
613
|
-- ACCESS MANAGEMENT (a stock 'default' with access_management=1 will do).
|
|
634
614
|
--
|
|
615
|
+
-- A house is a database; its rooms are three shared tables. Everyone granted on the
|
|
616
|
+
-- database writes into the same tables, and every row carries who (user_id, stamped by
|
|
617
|
+
-- the server) and where from (host, the install's fingerprint). Adding a housemate later
|
|
618
|
+
-- is the same two statements: CREATE USER, GRANT ALL ON the house.
|
|
619
|
+
--
|
|
635
620
|
-- Every name is qualified and there is no USE, so this runs anywhere: clickhouse-client,
|
|
636
621
|
-- the play UI, curl, a GUI. Order matters only in that the database comes first.
|
|
637
622
|
--
|
|
@@ -655,31 +640,16 @@ CREATE USER IF NOT EXISTS ${member} IDENTIFIED BY '${password.replace(/'/g, "\\'
|
|
|
655
640
|
|
|
656
641
|
${rooms.trim()}
|
|
657
642
|
|
|
658
|
-
--
|
|
659
|
-
--
|
|
660
|
-
|
|
661
|
-
|
|
662
|
-
|
|
663
|
-
|
|
664
|
-
--
|
|
665
|
-
--
|
|
666
|
-
|
|
667
|
-
|
|
668
|
-
${mergeGrants}
|
|
669
|
-
|
|
670
|
-
-- Resource ceilings. Confidentiality isolation is a grant that is absent; this is the
|
|
671
|
-
-- availability half, and it did not exist until 0.4.7 — a member could raise
|
|
672
|
-
-- max_memory_usage, zero max_execution_time and take a shared server down for everyone.
|
|
673
|
-
-- CONST, not MAX: a plain default is advisory, and MAX is not enough either because 0 means
|
|
674
|
-
-- UNLIMITED in ClickHouse and 0 satisfies any MAX. Measured — with a MAX ceiling in force,
|
|
675
|
-
-- SETTINGS max_memory_usage = 0 was accepted and the ceiling was gone.
|
|
676
|
-
-- CREATE ... OR REPLACE, which this used, DETACHES every user already on the profile, so
|
|
677
|
-
-- provisioning a second member silently stripped the first member's ceilings. IF NOT
|
|
678
|
-
-- EXISTS + ALTER edits the same object in place and keeps every assignment.
|
|
679
|
-
CREATE SETTINGS PROFILE IF NOT EXISTS ${MEMBER_PROFILE};
|
|
680
|
-
ALTER SETTINGS PROFILE ${MEMBER_PROFILE} SETTINGS NONE;
|
|
681
|
-
ALTER SETTINGS PROFILE ${MEMBER_PROFILE} SETTINGS ${MEMBER_PROFILE_SETTINGS};
|
|
682
|
-
ALTER USER ${member} SETTINGS PROFILE '${MEMBER_PROFILE}';
|
|
643
|
+
-- The whole database. ALL on your own house reaches nothing outside it — the database is
|
|
644
|
+
-- the boundary — and it is what lets the shipper create and evolve its own tables.
|
|
645
|
+
GRANT ALL ON ${db}.* TO ${member};
|
|
646
|
+
|
|
647
|
+
-- Pin async_insert on the user (ADD SETTING merges; a bare SETTINGS clause would replace
|
|
648
|
+
-- the user's whole list). The pin keeps the user_id stamp honest: a MATERIALIZED
|
|
649
|
+
-- currentUser() is computed during the INSERT, and an async flush stores it as the empty
|
|
650
|
+
-- string. CONST refuses the override; a client that never mentions the setting is held
|
|
651
|
+
-- at 0 and its insert lands stamped.
|
|
652
|
+
ALTER USER ${member} ADD SETTING ${MEMBER_PIN};
|
|
683
653
|
`;
|
|
684
654
|
}
|
|
685
655
|
|
|
@@ -707,22 +677,19 @@ async function adminBootstrap(cfg, admin) {
|
|
|
707
677
|
catch { console.log(bad(`house '${cfg.db}' does not exist and '${admin.user}' cannot create it: ${e.message}`)); return null; }
|
|
708
678
|
}
|
|
709
679
|
|
|
710
|
-
// 2. the user.
|
|
711
|
-
//
|
|
712
|
-
//
|
|
713
|
-
//
|
|
714
|
-
// SELECT ON system.users. That is the isolation working — but the raw 497 names neither
|
|
715
|
-
// side, while INSTALL.md promises install "refuses, names both sides, and creates
|
|
716
|
-
// nothing". Say who you are not, and what the owner has to run instead.
|
|
680
|
+
// 2. the user. A shared house makes an existing user unremarkable — rows are told apart
|
|
681
|
+
// by the server-stamped user_id, so "the handle is taken" hands nobody anything. What
|
|
682
|
+
// still matters is proof: connecting AS the member below is what validates whichever
|
|
683
|
+
// password is in play, existing user or fresh one.
|
|
717
684
|
let exists;
|
|
718
685
|
try {
|
|
719
686
|
exists = (await chRows(adminCfg, `SELECT name FROM system.users WHERE name = '${admin.member}'`, { database: '' })).length > 0;
|
|
720
687
|
} catch (e) {
|
|
721
688
|
if (/Not enough privileges|ACCESS_DENIED|system\.users/i.test(e.message || '')) {
|
|
722
689
|
console.log(bad(`'${admin.user}' is not an admin of this house — it cannot read system.users, so it cannot provision a member`));
|
|
723
|
-
console.log('
|
|
724
|
-
console.log(`
|
|
725
|
-
console.log(' If you already HAVE a
|
|
690
|
+
console.log(' Joining a house is two statements for whoever owns it:');
|
|
691
|
+
console.log(` CREATE USER ${admin.member} IDENTIFIED BY '…'; GRANT ALL ON ${cfg.db}.* TO ${admin.member};`);
|
|
692
|
+
console.log(' If you already HAVE a credential on this house, you do not need admin:');
|
|
726
693
|
console.log(` memhouse install --url ${adminCfg.url} --user ${admin.member} --password …`);
|
|
727
694
|
return null;
|
|
728
695
|
}
|
|
@@ -730,19 +697,14 @@ async function adminBootstrap(cfg, admin) {
|
|
|
730
697
|
}
|
|
731
698
|
let password = flags['member-password'] || null;
|
|
732
699
|
if (exists) {
|
|
733
|
-
if (
|
|
734
|
-
console.log(bad(`ClickHouse user '${admin.member}' already exists
|
|
735
|
-
console.log(
|
|
736
|
-
console.log(' currentUser(), so their rows and yours would be indistinguishable.');
|
|
737
|
-
console.log(' If this is you on a new machine, prove it:');
|
|
738
|
-
console.log(` memhouse install --adopt-user --member ${admin.member} --member-password '…' …`);
|
|
739
|
-
console.log(` If it is someone else, pick another handle: --member <name>`);
|
|
700
|
+
if (!password) {
|
|
701
|
+
console.log(bad(`ClickHouse user '${admin.member}' already exists — pass --member-password to install as them.`));
|
|
702
|
+
console.log(` Or pick another handle: --member <name>`);
|
|
740
703
|
return null;
|
|
741
704
|
}
|
|
742
|
-
if (!password) { console.log(bad('--adopt-user requires --member-password — the password is the proof')); return null; }
|
|
743
705
|
try { await ch({ ...cfg, user: admin.member, password }, 'SELECT 1', { database: '' }); }
|
|
744
|
-
catch { console.log(bad(
|
|
745
|
-
console.log(ok(`
|
|
706
|
+
catch { console.log(bad(`'${admin.member}' did not authenticate with that password`)); return null; }
|
|
707
|
+
console.log(ok(`using the existing ClickHouse user '${admin.member}' (password verified)`));
|
|
746
708
|
} else {
|
|
747
709
|
if (!password) {
|
|
748
710
|
password = generatePassword();
|
|
@@ -758,33 +720,18 @@ async function adminBootstrap(cfg, admin) {
|
|
|
758
720
|
createdUser = admin.member;
|
|
759
721
|
}
|
|
760
722
|
|
|
761
|
-
// 3.
|
|
762
|
-
//
|
|
763
|
-
|
|
764
|
-
|
|
765
|
-
|
|
766
|
-
|
|
767
|
-
|
|
768
|
-
|
|
769
|
-
|
|
770
|
-
// REFUSAL, so a first-time installer reads a success as a warning about someone else.
|
|
771
|
-
const provisionArgs = [provision, '--member', admin.member, '--merge'];
|
|
772
|
-
if (createdUser) provisionArgs.push('--user-just-created');
|
|
773
|
-
const rc = spawnSync(process.execPath, provisionArgs, {
|
|
774
|
-
stdio: 'inherit',
|
|
775
|
-
env: { ...process.env, MEM_URL: cfg.url, MEM_USER: admin.user, MEM_PASSWORD: admin.password, MEM_DB: cfg.db },
|
|
776
|
-
});
|
|
777
|
-
if (rc.status !== 0) {
|
|
778
|
-
// "nothing was written" was a lie whenever the CREATE USER above had already run —
|
|
779
|
-
// and it runs before this. Observed: provision.js died on a missing dependency, this
|
|
780
|
-
// printed the all-clear, and the next install refused with the identity-takeover
|
|
781
|
-
// message because the user it said it had not created was there. Say what exists, and
|
|
782
|
-
// name the two ways forward.
|
|
783
|
-
console.log(bad('provisioning failed'));
|
|
723
|
+
// 3. the grant and the pin. GRANT ALL on the house is the whole tenancy model: it
|
|
724
|
+
// reaches nothing outside the database, and it is what lets the member's own shipper
|
|
725
|
+
// create and evolve the rooms (--ensure-schema below).
|
|
726
|
+
try {
|
|
727
|
+
await q(`GRANT ALL ON ${cfg.db}.* TO ${admin.member}`, { database: '' });
|
|
728
|
+
await q(`ALTER USER ${admin.member} ADD SETTING ${MEMBER_PIN}`, { database: '' });
|
|
729
|
+
console.log(ok(`granted the house: ALL ON ${cfg.db}.* to '${admin.member}', async_insert pinned`));
|
|
730
|
+
} catch (e) {
|
|
731
|
+
console.log(bad(`could not grant the house to '${admin.member}': ${e.message}`));
|
|
784
732
|
if (createdUser) {
|
|
785
|
-
console.log(` ClickHouse user '${createdUser}' WAS created before this failed
|
|
786
|
-
console.log(
|
|
787
|
-
console.log(` memhouse install --adopt-user --member ${createdUser} --member-password '<the password above>' …`);
|
|
733
|
+
console.log(` ClickHouse user '${createdUser}' WAS created before this failed.`);
|
|
734
|
+
console.log(` Continue once fixed: memhouse install --member ${createdUser} --member-password '<the password above>' …`);
|
|
788
735
|
console.log(` Or undo it as the admin: DROP USER ${createdUser}`);
|
|
789
736
|
} else {
|
|
790
737
|
console.log(' nothing was written');
|
|
@@ -793,11 +740,15 @@ async function adminBootstrap(cfg, admin) {
|
|
|
793
740
|
}
|
|
794
741
|
|
|
795
742
|
// The step that makes this trustworthy: stop being admin, and prove the credential we
|
|
796
|
-
// are about to persist
|
|
743
|
+
// are about to persist can build and reach the rooms itself.
|
|
797
744
|
const memberCfg = { ...cfg, user: admin.member, password };
|
|
745
|
+
if (run(SHIP_JS, ['--ensure-schema'], memberCfg) !== 0) {
|
|
746
|
+
console.log(bad(`'${admin.member}' could not create the rooms in '${cfg.db}' — nothing written to disk`));
|
|
747
|
+
return null;
|
|
748
|
+
}
|
|
798
749
|
try {
|
|
799
|
-
const seen = await chRows(memberCfg, `SELECT count() AS n FROM system.tables WHERE database = '${cfg.db}' AND name IN ('
|
|
800
|
-
if (Number(seen[0]?.n) !== 3) { console.log(bad(`'${admin.member}' cannot see
|
|
750
|
+
const seen = await chRows(memberCfg, `SELECT count() AS n FROM system.tables WHERE database = '${cfg.db}' AND name IN ('sessions','messages','tool_calls')`, { database: '' });
|
|
751
|
+
if (Number(seen[0]?.n) !== 3) { console.log(bad(`'${admin.member}' cannot see the three rooms — nothing written`)); return null; }
|
|
801
752
|
} catch (e) { console.log(bad(`'${admin.member}' could not connect after provisioning: ${e.message}`)); return null; }
|
|
802
753
|
console.log(ok(`verified as '${admin.member}' — admin credential discarded, not stored`));
|
|
803
754
|
return memberCfg;
|
|
@@ -855,6 +806,16 @@ async function cmdInstall({ interactive }) {
|
|
|
855
806
|
let cfg = resolveConfig();
|
|
856
807
|
const adminUser = flags['admin-user'];
|
|
857
808
|
|
|
809
|
+
// The house name is spliced unquoted into CREATE DATABASE and GRANT ALL ON <db>.* —
|
|
810
|
+
// validate it HERE, on every install mode, not only where deploy happens to choose it.
|
|
811
|
+
// Found by the acceptance run, driven end to end: `install --db system` sailed through,
|
|
812
|
+
// granted a member CHECK/SHOW/SELECT/INSERT/ALTER/DROP/TRUNCATE/… ON system.* and
|
|
813
|
+
// created sessions/messages/tool_calls INSIDE the server's system database, exit 0.
|
|
814
|
+
// On a shared or kernel ClickHouse that is a granted-everything-on-system user minted
|
|
815
|
+
// by a memhouse one-liner.
|
|
816
|
+
try { require(path.join(REPO_ROOT, 'memhouse', 'house', 'house')).assertUsableName(cfg.db, 'house'); }
|
|
817
|
+
catch (e) { console.log(bad(e.message)); return 1; }
|
|
818
|
+
|
|
858
819
|
// Option 1 — print the SQL and stop. For the common case: you have admin on this
|
|
859
820
|
// ClickHouse and would rather run four statements yourself than hand a credential to an
|
|
860
821
|
// installer. Nothing is written and nothing is contacted.
|
|
@@ -893,7 +854,7 @@ async function cmdInstall({ interactive }) {
|
|
|
893
854
|
if (keyProblem) {
|
|
894
855
|
console.log(bad(`the rooms are here, but their sorting keys are wrong: ${keyProblem}`));
|
|
895
856
|
console.log(' a ship pass would corrupt them, so `memhouse ship` will refuse.');
|
|
896
|
-
console.log(` rebuild them from ${path.join(REPO_ROOT, 'memhouse', '
|
|
857
|
+
console.log(` rebuild them from ${path.join(REPO_ROOT, 'memhouse', 'house', 'schema.sql.tpl')}, then: memhouse ship --full`);
|
|
897
858
|
console.log(' no config was written — nothing here reads as installed.');
|
|
898
859
|
return 1;
|
|
899
860
|
}
|
|
@@ -907,8 +868,8 @@ async function cmdInstall({ interactive }) {
|
|
|
907
868
|
}
|
|
908
869
|
if (flags['no-ship'] !== true && run(SHIP_JS, [], cfg) !== 0) return 1;
|
|
909
870
|
console.log(ok('installed — next: memhouse start (dashboard + shipper loop)'));
|
|
910
|
-
console.log('
|
|
911
|
-
console.log(`
|
|
871
|
+
console.log(' adding a housemate later is two statements for the admin:');
|
|
872
|
+
console.log(` CREATE USER <name> IDENTIFIED BY '…'; GRANT ALL ON ${cfg.db}.* TO <name>;`);
|
|
912
873
|
return 0;
|
|
913
874
|
}
|
|
914
875
|
|
|
@@ -917,7 +878,7 @@ async function cmdInstall({ interactive }) {
|
|
|
917
878
|
// authenticating as memhouse_root instead: the three flags did nothing, --yes satisfied
|
|
918
879
|
// haveAll so no prompt asked who you were, and resolveConfig's default was used AS A
|
|
919
880
|
// CREDENTIAL. Refuse rather than do something else silently.
|
|
920
|
-
const adminOnly = ['
|
|
881
|
+
const adminOnly = ['member', 'member-password'].filter((k) => flags[k] !== undefined);
|
|
921
882
|
if (!adminUser && adminOnly.length) {
|
|
922
883
|
console.log(bad(`--${adminOnly.join(', --')} ${adminOnly.length > 1 ? 'are' : 'is'} only read with --admin-user`));
|
|
923
884
|
console.log(' Those flags provision a member, which needs house admin. Without them this');
|
|
@@ -1030,7 +991,7 @@ async function cmdInstall({ interactive }) {
|
|
|
1030
991
|
if (keyProblem) {
|
|
1031
992
|
console.log(bad(`the rooms are here, but their sorting keys are wrong: ${keyProblem}`));
|
|
1032
993
|
console.log(' a ship pass would corrupt them, so `memhouse ship` will refuse.');
|
|
1033
|
-
console.log(` rebuild them from ${path.join(REPO_ROOT, 'memhouse', '
|
|
994
|
+
console.log(` rebuild them from ${path.join(REPO_ROOT, 'memhouse', 'house', 'schema.sql.tpl')}, then: memhouse ship --full`);
|
|
1034
995
|
console.log(' no config was written — nothing here reads as installed.');
|
|
1035
996
|
return 1;
|
|
1036
997
|
}
|
|
@@ -1320,7 +1281,7 @@ async function cmdDoctor() {
|
|
|
1320
1281
|
add(t === objects.length, t === objects.length
|
|
1321
1282
|
? `schema: 3/3 rooms in '${cfg.db}' (${objects.join(', ')})`
|
|
1322
1283
|
: `schema: ${t}/${objects.length} rooms in '${cfg.db}' — missing ${absent.join(', ')}`,
|
|
1323
|
-
`run: memhouse install
|
|
1284
|
+
`run: memhouse install — with ALL on the house, ship --ensure-schema builds them`);
|
|
1324
1285
|
} catch (e) { add(false, 'schema check', `${e.message} — run: memhouse install`); }
|
|
1325
1286
|
// Columns, not just rooms. A room with a column missing accepts every insert and
|
|
1326
1287
|
// discards that field — measured, 85 rows shipped with the value thrown away while
|
|
@@ -1328,7 +1289,7 @@ async function cmdDoctor() {
|
|
|
1328
1289
|
// counted rooms.
|
|
1329
1290
|
try {
|
|
1330
1291
|
const { templateColumns } = require(path.join(REPO_ROOT, 'memhouse', 'shipper', 'ship.js'));
|
|
1331
|
-
const tpl = fs.readFileSync(path.join(REPO_ROOT, 'memhouse', '
|
|
1292
|
+
const tpl = fs.readFileSync(path.join(REPO_ROOT, 'memhouse', 'house', 'schema.sql.tpl'), 'utf-8');
|
|
1332
1293
|
const want = templateColumns(tpl);
|
|
1333
1294
|
// Names AND types AND the MATERIALIZED kind. Comparing names alone left every other
|
|
1334
1295
|
// kind of drift invisible, with measured consequences: a `UInt64` column narrowed to
|
|
@@ -1393,7 +1354,7 @@ async function cmdDoctor() {
|
|
|
1393
1354
|
add(false, `sorting keys: only ${checked}/${ROOM_TYPES.length} rooms are MergeTree — the rest have no sorting key at all`,
|
|
1394
1355
|
'a room was replaced by a Merge/View/Log engine; rebuild it from the schema template as the house owner');
|
|
1395
1356
|
} else add(wrongKeys.length === 0, `sorting keys${wrongKeys.length ? `: wrong on ${wrongKeys.join(', ')}` : ` carry origin correctly (${checked}/${ROOM_TYPES.length} rooms)`}`,
|
|
1396
|
-
`rebuild those rooms from the schema template, then: memhouse ship --full\n template: ${path.join(REPO_ROOT, 'memhouse', '
|
|
1357
|
+
`rebuild those rooms from the schema template, then: memhouse ship --full\n template: ${path.join(REPO_ROOT, 'memhouse', 'house', 'schema.sql.tpl')}`);
|
|
1397
1358
|
} catch (e) { add(false, 'sorting keys', e.message); }
|
|
1398
1359
|
try {
|
|
1399
1360
|
// countIf, not any(). `any()` returns an arbitrary row's value, so on a house with
|
|
@@ -1931,8 +1892,8 @@ async function cmdReset() {
|
|
|
1931
1892
|
} catch { /* pre-origin house: nothing to protect, ensureSchema will add the column */ }
|
|
1932
1893
|
|
|
1933
1894
|
if (flags.yes !== true) {
|
|
1934
|
-
// Name the rooms
|
|
1935
|
-
//
|
|
1895
|
+
// Name the rooms, and the scope honestly: the tables are shared, but reset only ever
|
|
1896
|
+
// deletes rows whose user_id is the CALLER's — a housemate's rows are untouched.
|
|
1936
1897
|
const scope = allOrigins
|
|
1937
1898
|
? `EVERY row, including ${imported} imported one(s) that re-shipping CANNOT restore,`
|
|
1938
1899
|
: 'the rows the shipper wrote';
|
|
@@ -2271,10 +2232,43 @@ async function cmdUninstall() {
|
|
|
2271
2232
|
: process.env.MEMHOUSE_PASSWORD ? 'MEMHOUSE_PASSWORD' : null;
|
|
2272
2233
|
if (initialised && rotateAsk) {
|
|
2273
2234
|
console.log(bad(`${rotateAsk} cannot change the credential of an existing house — the image only applies it when it initialises the data directory.`));
|
|
2274
|
-
console.log(
|
|
2235
|
+
console.log(` to rotate: ALTER USER ${priorCfg.user || 'memhouse_root'} IDENTIFIED BY '…' inside the house, then: memhouse setup --password …`);
|
|
2275
2236
|
console.log(' to start over (DESTROYS the memory): memhouse deploy --down');
|
|
2276
2237
|
process.exitCode = 2; break;
|
|
2277
2238
|
}
|
|
2239
|
+
// Who owns this house. A local deploy is the pilot's own machine and their own
|
|
2240
|
+
// memory, so the house user is THEM — suggested from the OS username (`polat`, not
|
|
2241
|
+
// `memhouse_root`) — created by the container itself at first init with full
|
|
2242
|
+
// rights. No admin/member split, no provisioning dance: a URI and a credential is
|
|
2243
|
+
// the whole install. The name is fixed at volume init exactly like the password,
|
|
2244
|
+
// so an existing house keeps the user it was born with.
|
|
2245
|
+
const osName = (os.userInfo().username || '').replace(/[^A-Za-z0-9_]/g, '_').replace(/^[0-9_]+/, '');
|
|
2246
|
+
let houseUser;
|
|
2247
|
+
if (initialised) {
|
|
2248
|
+
// The env FILE, not resolveConfig(): flags outrank the file there, so comparing
|
|
2249
|
+
// `--user other` against priorCfg.user compares the flag against itself and the
|
|
2250
|
+
// refusal below can never fire — while the volume stays initialised with the
|
|
2251
|
+
// original name and every later connection fails auth.
|
|
2252
|
+
const storedUser = readEnvFile().MEMHOUSE_USER;
|
|
2253
|
+
// No stored user means the env file is gone or emptied — the stored PASSWORD is
|
|
2254
|
+
// gone with it, so the `initialised && !reusable` refusal below already owns this
|
|
2255
|
+
// case. Refusing `--user` here on a guessed 'memhouse_root' would tell the pilot
|
|
2256
|
+
// their house "was initialised with memhouse_root" on no evidence at all.
|
|
2257
|
+
if (storedUser && flags.user && flags.user !== true && flags.user !== storedUser) {
|
|
2258
|
+
console.log(bad(`--user cannot rename the user of an existing house — the image creates it only when it initialises the data directory.`));
|
|
2259
|
+
console.log(` this house was initialised with '${storedUser}'. To start over (DESTROYS the memory): memhouse deploy --down`);
|
|
2260
|
+
process.exitCode = 2; break;
|
|
2261
|
+
}
|
|
2262
|
+
houseUser = storedUser || 'memhouse_root';
|
|
2263
|
+
} else if (flags.user && flags.user !== true) {
|
|
2264
|
+
houseUser = String(flags.user);
|
|
2265
|
+
} else if (flags.yes !== true && process.stdin.isTTY) {
|
|
2266
|
+
houseUser = (await ask(`House user (this becomes your member name)`, osName || 'memhouse_root')).trim() || osName || 'memhouse_root';
|
|
2267
|
+
} else {
|
|
2268
|
+
houseUser = osName || 'memhouse_root';
|
|
2269
|
+
}
|
|
2270
|
+
try { require(path.join(REPO_ROOT, 'memhouse', 'house', 'house')).assertUsableName(houseUser, 'house user'); }
|
|
2271
|
+
catch (e) { console.log(bad(`'${houseUser}' cannot be the house user — ${e.message}`)); process.exitCode = 2; break; }
|
|
2278
2272
|
// An initialised volume with no credential to reuse — env file deleted, emptied, or
|
|
2279
2273
|
// never written — is the same lockout by another route: a generated password would
|
|
2280
2274
|
// be ignored by the server and then written over the config as if it worked.
|
|
@@ -2301,7 +2295,17 @@ async function cmdUninstall() {
|
|
|
2301
2295
|
const tag = flags.tag || process.env.MEMHOUSE_CH_TAG || dep.managedTag() || dep.DEFAULT_TAG;
|
|
2302
2296
|
// The database is part of the destination, not a detail of it — see the service
|
|
2303
2297
|
// check below, which must compare it.
|
|
2304
|
-
|
|
2298
|
+
// STATED sources only, then the house user. priorCfg.db is unusable here: resolveConfig
|
|
2299
|
+
// computes a DEFAULT for it (the user's name, itself defaulted), so going through it
|
|
2300
|
+
// handed a fresh deploy `memhouse_root` as the house name while the superuser was
|
|
2301
|
+
// 'polat' — the memory would live in a database named after a role account nobody
|
|
2302
|
+
// chose.
|
|
2303
|
+
const targetDb = flags.db || process.env.MEMHOUSE_DB || readEnvFile().MEMHOUSE_DB || houseUser;
|
|
2304
|
+
// Same check install runs, but EARLY — deploy initialises the container volume and
|
|
2305
|
+
// writes the env file before its install step, so a refusal that waits for install
|
|
2306
|
+
// arrives after the damage.
|
|
2307
|
+
try { require(path.join(REPO_ROOT, 'memhouse', 'house', 'house')).assertUsableName(targetDb, 'house'); }
|
|
2308
|
+
catch (e) { console.log(bad(e.message)); process.exitCode = 2; break; }
|
|
2305
2309
|
{
|
|
2306
2310
|
// A service-managed shipper keeps the environment it was installed with, so ANY
|
|
2307
2311
|
// switch that repoints the config leaves it shipping somewhere else — not only an
|
|
@@ -2355,7 +2359,7 @@ async function cmdUninstall() {
|
|
|
2355
2359
|
}
|
|
2356
2360
|
const pw = reusable || crypto.randomBytes(16).toString('hex');
|
|
2357
2361
|
if (reusable) console.log(ok('reusing the existing house credential (its data volume is already initialised)'));
|
|
2358
|
-
const r = dep.up({ password: pw, port, tag });
|
|
2362
|
+
const r = dep.up({ password: pw, port, tag, user: houseUser });
|
|
2359
2363
|
if (!r.ok) { console.log(bad(r.msg)); process.exitCode = 1; break; }
|
|
2360
2364
|
console.log(ok(`ClickHouse starting via ${r.engine} on ${r.url} (loopback only)`));
|
|
2361
2365
|
|
|
@@ -2367,7 +2371,10 @@ async function cmdUninstall() {
|
|
|
2367
2371
|
// house that eventually came up is unreachable forever. The image applies the
|
|
2368
2372
|
// password only at first init, so there is no way back from that.
|
|
2369
2373
|
if (!reusable) {
|
|
2370
|
-
|
|
2374
|
+
// db INCLUDED: this file is read back by the install below, and writing
|
|
2375
|
+
// priorCfg's computed default here made that install refuse over a mismatch
|
|
2376
|
+
// with a file this same command had just written.
|
|
2377
|
+
writeEnvFile({ ...priorCfg, url: r.url, user: houseUser, password: pw, db: targetDb });
|
|
2371
2378
|
console.log(ok(`credential saved to ${ENV_FILE} before waiting — the volume is initialised with it`));
|
|
2372
2379
|
}
|
|
2373
2380
|
|
|
@@ -2417,10 +2424,10 @@ async function cmdUninstall() {
|
|
|
2417
2424
|
// anticipates it) and it is NOT the narrow set INSTALL.md's grant-set argument
|
|
2418
2425
|
// describes. A user who later adds a second member should know which house they
|
|
2419
2426
|
// have.
|
|
2420
|
-
console.log(warn(`this house is yours alone: '${
|
|
2421
|
-
console.log(' Adding a
|
|
2422
|
-
console.log(`
|
|
2423
|
-
flags.url = r.url; flags.user =
|
|
2427
|
+
console.log(warn(`this house is yours alone: '${houseUser}' is its superuser, and that is the credential being saved`));
|
|
2428
|
+
console.log(' Adding a housemate later is two statements:');
|
|
2429
|
+
console.log(` CREATE USER <name> IDENTIFIED BY '…'; GRANT ALL ON ${targetDb}.* TO <name>;`);
|
|
2430
|
+
flags.url = r.url; flags.user = houseUser; flags.password = pw;
|
|
2424
2431
|
flags.db = targetDb;
|
|
2425
2432
|
flags.yes = true;
|
|
2426
2433
|
process.exitCode = await cmdInstall({ interactive: false });
|
package/memhouse/DESIGN.md
CHANGED
|
@@ -5,10 +5,12 @@ on your machines — across **all 17 editors** agentlytics supports — parsed l
|
|
|
5
5
|
shipped to a typed ClickHouse store, shareable with a team, installable on the
|
|
6
6
|
ultimagent kernel as an **agency**, and visible through the agentlytics dashboard.
|
|
7
7
|
|
|
8
|
-
**
|
|
9
|
-
`
|
|
10
|
-
|
|
11
|
-
|
|
8
|
+
**A house is a database; its rooms are three shared tables** — `sessions`, `messages`,
|
|
9
|
+
`tool_calls`. Everyone in the house writes into the same tables with their own
|
|
10
|
+
credential; the server-stamped `user_id` says who, the install fingerprint `host` says
|
|
11
|
+
which machine. Sharing IS the house; the boundary between groups is the database. (The
|
|
12
|
+
layout has moved twice: shared-tables-with-row-policies fell in 0.4.0, per-member
|
|
13
|
+
suffixed rooms fell in 0.8.0 — see `house/HOUSE.md` for why.)
|
|
12
14
|
|
|
13
15
|
**Agency, precisely.** In constellation terms (`../TERMINOLOGY.md`) memhouse is a
|
|
14
16
|
**house** — the `mem` database — **plus a resident**: the shipper. That pairing
|
|
@@ -24,23 +26,21 @@ on = "loop" # a daemon — not an insert trigger, not a schedule
|
|
|
24
26
|
```
|
|
25
27
|
|
|
26
28
|
The session rollup is a **routine** — house machinery, not a second resident. Since
|
|
27
|
-
0.4.0 it is not even an object: it is SQL text (`
|
|
28
|
-
|
|
29
|
-
nothing. memhouse has **no materialized views at all** — in fact no stored views at all
|
|
29
|
+
0.4.0 it is not even an object: it is SQL text (`house/house.js`), computed during your
|
|
30
|
+
query, for your query, writing nothing. memhouse has **no materialized views at all** — in fact no stored views at all
|
|
30
31
|
— so the shipper is not merely a resident, it is the only candidate in the tree. Strip
|
|
31
32
|
it and every remaining moving part is a routine over rows nobody is writing any more.
|
|
32
33
|
|
|
33
34
|
Note that `user_id MATERIALIZED currentUser()` is a materialized *column*: it computes
|
|
34
|
-
during your own insert and is part of the table's definition. It is
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
property, not what makes this an agency.
|
|
35
|
+
during your own insert and is part of the table's definition. It is what tells housemates'
|
|
36
|
+
rows apart in the shared tables — provenance, stamped by the server, with async_insert
|
|
37
|
+
pinned so it cannot be skipped. An identity property, not what makes this an agency.
|
|
38
38
|
|
|
39
39
|
**Positioning:** an alternative agency **competing with memory-house**. It borrows
|
|
40
40
|
memory-house's proven ideas (server-stamped identity, idempotent shipping,
|
|
41
41
|
skills/plugin delivery) and agentlytics' proven assets (adapters, UI, cost engine) —
|
|
42
|
-
but **not** its
|
|
43
|
-
one
|
|
42
|
+
but **not** its row-policy machinery: memhouse's tables are shared by people who chose
|
|
43
|
+
one house, and the boundary between houses is the database itself. If it wins, it can become memory-house v4; until then the
|
|
44
44
|
two run side by side as separate agencies, each in its own house. Naming: the product is
|
|
45
45
|
**memhouse**; the agency is **`memhouse`** and its house is the **`mem`** database
|
|
46
46
|
(identifiers can't carry a dash; the owner kept the longer name, `memhouse_root`).
|
|
@@ -60,20 +60,19 @@ them free and further along; **bet 3 is the one nobody else attempts.**
|
|
|
60
60
|
write axis this bet is precisely *moving work from routine to resident*:
|
|
61
61
|
memory-house parses when you query, memhouse parses before anyone asks and
|
|
62
62
|
writes the result down.
|
|
63
|
-
2. **Typed common schema** (`
|
|
64
|
-
|
|
65
|
-
memory-house
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
3. **Kernel-installable agency.**
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
themselves (grant-option) or the owner.
|
|
63
|
+
2. **Typed common schema** (`house/schema.sql.tpl` — three shared tables per house).
|
|
64
|
+
Physical typed columns (what memory-house derives in views) + `tool_calls`
|
|
65
|
+
(memory-house has no tool table) + one `extra JSON` escape hatch per room so
|
|
66
|
+
unnormalized adapter fields are never lost. `ReplacingMergeTree(ingested_at)`;
|
|
67
|
+
`messages` keyed `(session_id, user_id, origin, seq)` so a re-ship of a
|
|
68
|
+
grown/changed session replaces stale rows (latest-wins) and two housemates' rows
|
|
69
|
+
never collapse into one. FTS text indexes built in (CH ≥ 26.2).
|
|
70
|
+
3. **Kernel-installable agency.** Joining any ClickHouse — a kernel's included — is a
|
|
71
|
+
database plus a credential: `CREATE USER`, `GRANT ALL ON <house>.* `, done. The
|
|
72
|
+
shipper is the resident that lands with it and creates its own rooms. Identity is
|
|
73
|
+
`user_id MATERIALIZED currentUser()` with `async_insert = 0 CONST` pinned on the
|
|
74
|
+
user; machines are told apart by the `host` fingerprint. The boundary between
|
|
75
|
+
agencies on one server is the database — a house's ALL reaches nothing outside it.
|
|
77
76
|
4. **Borrowed UI, zero fork.** The React SPA consumes REST JSON, not tables. The
|
|
78
77
|
memhouse server implements the **same `/api/*` contract** as agentlytics'
|
|
79
78
|
`server.js` (same routes, same response shapes) over the memhouse schema, and
|
|
@@ -83,9 +82,9 @@ them free and further along; **bet 3 is the one nobody else attempts.**
|
|
|
83
82
|
|
|
84
83
|
| Path | What | Notes |
|
|
85
84
|
|---|---|---|
|
|
86
|
-
| `
|
|
87
|
-
| `
|
|
88
|
-
| `
|
|
85
|
+
| `house/house.js` | room names + the session rollup + the user pin | one rule for both transports |
|
|
86
|
+
| `house/schema.sql.tpl` | the house's three shared tables (typed) | applied by each member's own shipper |
|
|
87
|
+
| `house/HOUSE.md` | the model, and what it replaced | |
|
|
89
88
|
| `shipper/ship.js` | parse-on-client shipper CLI — the resident (`worker`) | reuses `../../editors`; incremental; idempotent |
|
|
90
89
|
| `server/server.js` | REST API + dashboard | same API contract as agentlytics; serves `../../public` |
|
|
91
90
|
| `delivery/` | delivery kit | installer, skills, plugin, AGENT-INSTALL.md, prompt |
|