@tpsdev-ai/flair 0.30.0 → 0.31.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +194 -377
- package/dist/cli.js +1434 -288
- package/dist/deploy.js +212 -24
- package/dist/fabric-upgrade.js +16 -1
- package/dist/federation/scheduler.js +500 -0
- package/dist/install/clients.js +111 -53
- package/dist/lib/mcp-spec.js +128 -0
- package/dist/lib/safe-snapshot-extract.js +231 -0
- package/dist/lib/scheduler-platform.js +128 -0
- package/dist/lib/xml-escape.js +54 -0
- package/dist/rem/scheduler.js +35 -87
- package/dist/rem/snapshot.js +13 -0
- package/dist/replication-convergence.js +505 -0
- package/dist/resources/AdminPrincipals.js +10 -1
- package/dist/resources/Agent.js +105 -11
- package/dist/resources/AgentSeed.js +10 -2
- package/dist/resources/MemoryBootstrap.js +7 -8
- package/dist/resources/MemoryUsage.js +18 -0
- package/dist/resources/Presence.js +8 -1
- package/dist/resources/SemanticSearch.js +17 -45
- package/dist/resources/abstention.js +1 -1
- package/dist/resources/agent-admin.js +149 -0
- package/dist/resources/agent-auth.js +98 -5
- package/dist/resources/auth-middleware.js +92 -19
- package/dist/resources/embeddings-boot.js +10 -12
- package/dist/resources/embeddings-provider.js +10 -7
- package/dist/resources/health.js +24 -19
- package/dist/resources/in-process.js +234 -0
- package/dist/resources/mcp-handler.js +14 -4
- package/dist/resources/mcp-tools.js +23 -17
- package/dist/resources/migration-boot.js +80 -10
- package/dist/resources/migrations/data-dir.js +205 -0
- package/dist/resources/migrations/progress.js +33 -0
- package/dist/resources/migrations/runner.js +29 -2
- package/dist/resources/migrations/state.js +13 -2
- package/dist/resources/models-dir.js +18 -9
- package/dist/resources/presence-internal.js +6 -1
- package/dist/resources/record-owner-guard.js +149 -0
- package/dist/resources/semantic-retrieval-core.js +5 -4
- package/dist/src/lib/scheduler-platform.js +128 -0
- package/dist/src/lib/xml-escape.js +54 -0
- package/dist/src/rem/scheduler.js +35 -87
- package/docs/deploying-on-fabric.md +267 -0
- package/docs/deployment.md +5 -0
- package/docs/embedding-in-a-harper-app.md +303 -0
- package/docs/federation.md +61 -4
- package/docs/integrations.md +3 -0
- package/docs/mcp-clients.md +16 -7
- package/docs/quickstart.md +80 -54
- package/docs/releasing.md +72 -38
- package/docs/supply-chain-policy.md +36 -0
- package/docs/troubleshooting.md +24 -0
- package/docs/upgrade.md +99 -4
- package/package.json +1 -11
- package/templates/bin/flair-federation-sync.sh.tmpl +28 -0
- package/templates/launchd/dev.flair.federation.sync.plist.tmpl +47 -0
- package/templates/systemd/flair-federation-sync.service.tmpl +21 -0
- package/templates/systemd/flair-federation-sync.timer.tmpl +16 -0
- package/dist/resources/rerank-provider.js +0 -569
- package/docs/rerank-provisioning.md +0 -101
|
@@ -0,0 +1,205 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* data-dir.ts — resolves the directory the migration subsystem owns
|
|
3
|
+
* (`<dataDir>/.migrations/{state.json,lock,snapshots,exports}`), and proves
|
|
4
|
+
* it is actually usable BEFORE the boot cycle commits to it.
|
|
5
|
+
*
|
|
6
|
+
* ─── Why this file exists (flair#812) ──────────────────────────────────────
|
|
7
|
+
* The previous resolution was a single expression, duplicated in
|
|
8
|
+
* `resources/migration-boot.ts` and `resources/health.ts`:
|
|
9
|
+
*
|
|
10
|
+
* process.env.HDB_ROOT ?? join(homedir(), ".flair", "data")
|
|
11
|
+
*
|
|
12
|
+
* Both halves of that were wrong in a way that only shows up off the
|
|
13
|
+
* developer/default shape:
|
|
14
|
+
*
|
|
15
|
+
* 1. `HDB_ROOT` IS NEVER SET BY ANYTHING. It is not a Harper process env
|
|
16
|
+
* var — Harper's own root-path env var is `ROOTPATH`
|
|
17
|
+
* (`harper`'s `utility/common_utils.ts` reads
|
|
18
|
+
* `process.env['ROOTPATH']`; `HDB_ROOT` appears only as a legacy *config
|
|
19
|
+
* key* alias in `utility/hdbTerms.ts`, and Harper never reads or writes
|
|
20
|
+
* it on `process.env`). flair's own spawner (`src/cli.ts`) exports
|
|
21
|
+
* `ROOTPATH`, never `HDB_ROOT`, and `resources/models-dir.ts` already
|
|
22
|
+
* reads `ROOTPATH` for exactly this reason. So the `??` left branch was
|
|
23
|
+
* dead code and the effective resolution was UNCONDITIONALLY
|
|
24
|
+
* `~/.flair/data`, regardless of where Harper's real root actually is.
|
|
25
|
+
*
|
|
26
|
+
* 2. `~/.flair/data` is right only by coincidence — it is `flair init`'s
|
|
27
|
+
* default data dir, so on a default local install the two happen to
|
|
28
|
+
* coincide. On a PROVISIONED install (a service-managed spoke, a
|
|
29
|
+
* container, a Harper Fabric component deployment) the process's
|
|
30
|
+
* `homedir()` is whatever the service account has: possibly read-only,
|
|
31
|
+
* possibly nonexistent, and in general nothing to do with the instance's
|
|
32
|
+
* data.
|
|
33
|
+
*
|
|
34
|
+
* When `~/.flair/data` is not creatable, EVERY consumer of the old
|
|
35
|
+
* expression failed — and, critically, failed SILENTLY: the very first
|
|
36
|
+
* thing `runMigrationCycle` does is `acquireMigrationLock`, whose
|
|
37
|
+
* `mkdirSync(dirname(lockPath), { recursive: true, mode: 0o700 })` throws
|
|
38
|
+
* `EACCES`/`ENOENT`, which the runner catches and turns into
|
|
39
|
+
* `{ ran: false, reason: "lock error: ..." }` — a value `migration-boot.ts`
|
|
40
|
+
* then discarded. No log line, no state file, no health signal, and every
|
|
41
|
+
* migration ever shipped skipped forever on that instance. (Contrast
|
|
42
|
+
* `resources/embeddings-boot.ts`, loaded by the SAME `jsResource` glob:
|
|
43
|
+
* it touches no filesystem path of its own, which is exactly why embeddings
|
|
44
|
+
* kept working on the affected shapes while migrations did not.)
|
|
45
|
+
*
|
|
46
|
+
* ─── The resolution ─────────────────────────────────────────────────────
|
|
47
|
+
* An ordered candidate list, first USABLE one wins:
|
|
48
|
+
*
|
|
49
|
+
* 1. `FLAIR_MIGRATION_DATA_DIR` — explicit operator override. The escape
|
|
50
|
+
* hatch for any deployment shape whose data dir this module cannot
|
|
51
|
+
* infer; naming it in the failure message below is what makes an
|
|
52
|
+
* unresolvable instance actionable rather than merely reported.
|
|
53
|
+
* 2. `HDB_ROOT` — retained purely for compatibility with anything that
|
|
54
|
+
* may have started setting it because flair once read it. Never set by
|
|
55
|
+
* Harper or by flair.
|
|
56
|
+
* 3. `homedir()/.flair/data` — the historical default, and `flair init`'s
|
|
57
|
+
* own data dir. DELIBERATELY still ahead of `ROOTPATH`: on every
|
|
58
|
+
* currently-working install this is where the state file already
|
|
59
|
+
* lives, and reordering would silently relocate it (costing a
|
|
60
|
+
* re-detect pass and orphaning the existing audit record) on instances
|
|
61
|
+
* that have no problem at all. This fix is about instances where the
|
|
62
|
+
* cycle cannot run, not about relocating ones where it can.
|
|
63
|
+
* 4. `ROOTPATH` — Harper's real root path. The rescue candidate: it is
|
|
64
|
+
* writable by definition on a running instance (Harper is writing its
|
|
65
|
+
* own databases there), so a shape whose `homedir()` is unusable still
|
|
66
|
+
* gets a stable, per-instance location instead of nothing.
|
|
67
|
+
* 5. The Harper root INFERRED from this module's own location, but only
|
|
68
|
+
* when the running layout is unambiguously a deployed component (see
|
|
69
|
+
* `deployedComponentRootPath`). `ROOTPATH` is an env var, and a
|
|
70
|
+
* deployment that configures `rootPath` in Harper's config file
|
|
71
|
+
* instead of the environment leaves candidate 4 empty — this covers
|
|
72
|
+
* that case without guessing.
|
|
73
|
+
*
|
|
74
|
+
* "Usable" is probed by DOING THE REAL OPERATION the runner would do —
|
|
75
|
+
* create `<dir>/.migrations` at 0700 and check it is writable — not by a
|
|
76
|
+
* proxy check that could disagree with it. The probe is idempotent and, on
|
|
77
|
+
* a healthy instance, is satisfied by the first candidate without touching
|
|
78
|
+
* the others.
|
|
79
|
+
*
|
|
80
|
+
* If NO candidate is usable, `resolveWritableMigrationDataDir` returns
|
|
81
|
+
* `dataDir: null` WITH the per-candidate reasons, and the boot path turns
|
|
82
|
+
* that into a loud console error plus a `failed` progress entry per
|
|
83
|
+
* registered migration — visible in `/HealthDetail`, `flair doctor` and
|
|
84
|
+
* `flair quality`'s `instance.migrationsClean`. An instance that cannot run
|
|
85
|
+
* migrations now says so; that silence was the actual defect.
|
|
86
|
+
*/
|
|
87
|
+
import { accessSync, constants, existsSync, mkdirSync } from "node:fs";
|
|
88
|
+
import { homedir } from "node:os";
|
|
89
|
+
import { basename, dirname, join } from "node:path";
|
|
90
|
+
import { fileURLToPath } from "node:url";
|
|
91
|
+
/** Explicit operator override for the migration data dir (see module doc). */
|
|
92
|
+
export const MIGRATION_DATA_DIR_ENV = "FLAIR_MIGRATION_DATA_DIR";
|
|
93
|
+
/** The subdirectory the migration subsystem owns inside whichever dataDir wins. */
|
|
94
|
+
export const MIGRATIONS_SUBDIR = ".migrations";
|
|
95
|
+
/**
|
|
96
|
+
* Harper's root path, inferred from where THIS module is running, but ONLY
|
|
97
|
+
* when the layout is unambiguously a deployed component:
|
|
98
|
+
*
|
|
99
|
+
* <rootPath>/components/<name>/dist/resources/migrations/data-dir.js
|
|
100
|
+
*
|
|
101
|
+
* i.e. the directory two levels above `dist/` must itself be named
|
|
102
|
+
* `components` — Harper's `componentsRoot` convention, and the layout
|
|
103
|
+
* `deploy_component` produces. That check is what keeps this from firing on
|
|
104
|
+
* a source checkout or an npm install, where the same arithmetic would
|
|
105
|
+
* point at an arbitrary parent directory. Returns null whenever the layout
|
|
106
|
+
* doesn't match, so a non-match contributes no candidate rather than a
|
|
107
|
+
* guess.
|
|
108
|
+
*
|
|
109
|
+
* `moduleUrl` is a parameter purely so a unit test can exercise both the
|
|
110
|
+
* matching and non-matching layouts without relocating the built file.
|
|
111
|
+
*/
|
|
112
|
+
export function deployedComponentRootPath(moduleUrl = import.meta.url) {
|
|
113
|
+
try {
|
|
114
|
+
const here = dirname(fileURLToPath(moduleUrl)); // <component>/dist/resources/migrations
|
|
115
|
+
const componentDir = dirname(dirname(dirname(here))); // <component>
|
|
116
|
+
const componentsRoot = dirname(componentDir); // .../components
|
|
117
|
+
if (basename(componentsRoot) !== "components")
|
|
118
|
+
return null;
|
|
119
|
+
return dirname(componentsRoot);
|
|
120
|
+
}
|
|
121
|
+
catch {
|
|
122
|
+
return null;
|
|
123
|
+
}
|
|
124
|
+
}
|
|
125
|
+
/**
|
|
126
|
+
* Ordered candidates for the migration data dir. See the module doc for why
|
|
127
|
+
* this order is what it is — in particular, why `homedir()/.flair/data`
|
|
128
|
+
* stays ahead of `ROOTPATH`.
|
|
129
|
+
*/
|
|
130
|
+
export function migrationDataDirCandidates(env = process.env, moduleUrl = import.meta.url) {
|
|
131
|
+
const out = [];
|
|
132
|
+
const add = (p) => {
|
|
133
|
+
const trimmed = p?.trim();
|
|
134
|
+
if (trimmed && !out.includes(trimmed))
|
|
135
|
+
out.push(trimmed);
|
|
136
|
+
};
|
|
137
|
+
add(env[MIGRATION_DATA_DIR_ENV]);
|
|
138
|
+
add(env.HDB_ROOT);
|
|
139
|
+
add(join(homedir(), ".flair", "data"));
|
|
140
|
+
add(env.ROOTPATH);
|
|
141
|
+
add(deployedComponentRootPath(moduleUrl));
|
|
142
|
+
return out;
|
|
143
|
+
}
|
|
144
|
+
/**
|
|
145
|
+
* Probes a candidate by performing the exact operation the runner's lock
|
|
146
|
+
* acquisition performs (`mkdir -p <dir>/.migrations` at 0700), then
|
|
147
|
+
* confirming the result is writable. Idempotent: on an already-working
|
|
148
|
+
* instance this is a no-op stat/mkdir against a directory that already
|
|
149
|
+
* exists.
|
|
150
|
+
*/
|
|
151
|
+
export function probeMigrationDataDir(dir) {
|
|
152
|
+
const owned = join(dir, MIGRATIONS_SUBDIR);
|
|
153
|
+
try {
|
|
154
|
+
mkdirSync(owned, { recursive: true, mode: 0o700 });
|
|
155
|
+
accessSync(owned, constants.W_OK | constants.X_OK);
|
|
156
|
+
return { dir, ok: true };
|
|
157
|
+
}
|
|
158
|
+
catch (err) {
|
|
159
|
+
return { dir, ok: false, reason: err?.message ?? String(err) };
|
|
160
|
+
}
|
|
161
|
+
}
|
|
162
|
+
/**
|
|
163
|
+
* Resolves a migration data dir that is proven writable, or reports why not.
|
|
164
|
+
* Used by the boot path (`resources/migration-boot.ts`), which must not
|
|
165
|
+
* commit to a directory the runner will then fail to lock.
|
|
166
|
+
*/
|
|
167
|
+
export function resolveWritableMigrationDataDir(env = process.env) {
|
|
168
|
+
const tried = [];
|
|
169
|
+
for (const candidate of migrationDataDirCandidates(env)) {
|
|
170
|
+
const probe = probeMigrationDataDir(candidate);
|
|
171
|
+
tried.push(probe);
|
|
172
|
+
if (probe.ok)
|
|
173
|
+
return { dataDir: candidate, tried };
|
|
174
|
+
}
|
|
175
|
+
return { dataDir: null, tried };
|
|
176
|
+
}
|
|
177
|
+
/**
|
|
178
|
+
* Operator-facing explanation of a total resolution failure — carries the
|
|
179
|
+
* actor (each path tried), the state (why each was rejected) and the remedy
|
|
180
|
+
* (the override env var), so the message is actionable at 3am rather than
|
|
181
|
+
* merely accurate.
|
|
182
|
+
*/
|
|
183
|
+
export function describeUnresolvableDataDir(tried) {
|
|
184
|
+
const detail = tried.length
|
|
185
|
+
? tried.map((t) => `${t.dir} (${t.reason ?? "unusable"})`).join("; ")
|
|
186
|
+
: "no candidate paths at all";
|
|
187
|
+
return (`no writable migration data directory — tried ${detail}. ` +
|
|
188
|
+
`Migrations cannot run on this instance until one exists: point ${MIGRATION_DATA_DIR_ENV} ` +
|
|
189
|
+
`at a writable directory (or make one of the paths above writable by the account this ` +
|
|
190
|
+
`instance runs as) and restart.`);
|
|
191
|
+
}
|
|
192
|
+
/**
|
|
193
|
+
* READ-ONLY counterpart for `/HealthDetail`, which must never create
|
|
194
|
+
* directories as a side effect of a GET. Prefers whichever candidate
|
|
195
|
+
* already carries a `.migrations` directory (i.e. the one a boot actually
|
|
196
|
+
* chose), falling back to the first candidate so the field is never empty.
|
|
197
|
+
*/
|
|
198
|
+
export function resolveMigrationDataDirForRead(env = process.env) {
|
|
199
|
+
const candidates = migrationDataDirCandidates(env);
|
|
200
|
+
for (const candidate of candidates) {
|
|
201
|
+
if (existsSync(join(candidate, MIGRATIONS_SUBDIR)))
|
|
202
|
+
return candidate;
|
|
203
|
+
}
|
|
204
|
+
return candidates[0] ?? join(homedir(), ".flair", "data");
|
|
205
|
+
}
|
|
@@ -25,6 +25,39 @@ export function seedIdleProgress(ids) {
|
|
|
25
25
|
}
|
|
26
26
|
}
|
|
27
27
|
}
|
|
28
|
+
/**
|
|
29
|
+
* Marks every migration that never got off the starting line as `failed`
|
|
30
|
+
* with `reason` — and ONLY those (flair#812).
|
|
31
|
+
*
|
|
32
|
+
* The boot path calls this when a cycle couldn't run, so that a condition
|
|
33
|
+
* which is really instance-wide (no writable data dir, tables never ready,
|
|
34
|
+
* the lock itself unobtainable) shows up per-migration, which is the shape
|
|
35
|
+
* `flair doctor` and `flair quality`'s `instance.migrationsClean` actually
|
|
36
|
+
* read.
|
|
37
|
+
*
|
|
38
|
+
* Restricting it to `idle` entries is load-bearing, not tidiness: a cycle
|
|
39
|
+
* can fail AFTER some migrations already reached a terminal state — the
|
|
40
|
+
* pre-hash path, for instance, halts each candidate with its own precise
|
|
41
|
+
* reason and THEN reports cycle failure. Overwriting those would replace a
|
|
42
|
+
* specific `halted` (which the runner retries on the next boot, per its
|
|
43
|
+
* documented contract) with a blanket `failed`, and would flip migrations
|
|
44
|
+
* that genuinely completed this cycle to failed. Only an entry still `idle`
|
|
45
|
+
* is one nothing has said anything about yet.
|
|
46
|
+
*
|
|
47
|
+
* Returns the ids actually marked, so a caller can tell "nothing had
|
|
48
|
+
* started" from "the cycle died partway".
|
|
49
|
+
*/
|
|
50
|
+
export function markIdleMigrationsFailed(ids, reason) {
|
|
51
|
+
const marked = [];
|
|
52
|
+
for (const id of ids) {
|
|
53
|
+
const current = progressById.get(id);
|
|
54
|
+
if (current && current.state !== "idle")
|
|
55
|
+
continue;
|
|
56
|
+
progressById.set(id, { id, rowsDone: 0, rowsRemaining: 0, state: "failed", reason });
|
|
57
|
+
marked.push(id);
|
|
58
|
+
}
|
|
59
|
+
return marked;
|
|
60
|
+
}
|
|
28
61
|
export function _resetProgressForTests() {
|
|
29
62
|
progressById.clear();
|
|
30
63
|
cycleStatus = { phase: "idle" };
|
|
@@ -151,7 +151,21 @@ async function runCycleLocked(deps, lock) {
|
|
|
151
151
|
const candidates = [];
|
|
152
152
|
for (const migration of deps.registry.list()) {
|
|
153
153
|
if (isShortCircuited(state, migration.id, deps.runningVersion)) {
|
|
154
|
-
|
|
154
|
+
// flair#812: a short-circuit is an ASSERTION READ OFF A FILE, not an
|
|
155
|
+
// observation of the corpus — and that file is hand-editable (the
|
|
156
|
+
// documented remediation for a stuck migration is to correct it by
|
|
157
|
+
// hand). Reporting it as plain "completed" made a hand-written entry
|
|
158
|
+
// indistinguishable from a migration this boot actually verified.
|
|
159
|
+
// The skip itself is unchanged (it is the documented cheap path); it
|
|
160
|
+
// now just says what it is, so `flair doctor` never presents an
|
|
161
|
+
// unverified claim as a verified one.
|
|
162
|
+
setMigrationProgress({
|
|
163
|
+
id: migration.id,
|
|
164
|
+
rowsDone: 0,
|
|
165
|
+
rowsRemaining: 0,
|
|
166
|
+
state: "completed",
|
|
167
|
+
reason: `recorded complete at ${deps.runningVersion} in ${deps.statePath} — not re-verified this boot`,
|
|
168
|
+
});
|
|
155
169
|
continue;
|
|
156
170
|
}
|
|
157
171
|
setMigrationProgress({ id: migration.id, rowsDone: 0, rowsRemaining: 0, state: "checking" });
|
|
@@ -160,12 +174,18 @@ async function runCycleLocked(deps, lock) {
|
|
|
160
174
|
pending = await migration.detect();
|
|
161
175
|
}
|
|
162
176
|
catch (err) {
|
|
177
|
+
// flair#812: a throwing detect() removes this migration from the
|
|
178
|
+
// candidate set, so a cycle where EVERY detect() throws used to reach
|
|
179
|
+
// the `nothing pending` return and look like a clean no-op. Log it
|
|
180
|
+
// with the standard marker as well as recording it in progress.
|
|
181
|
+
const reason = `detect() threw: ${err?.message ?? String(err)}`;
|
|
182
|
+
console.error(`[flair-migrations] ${migration.id}: ${reason}`);
|
|
163
183
|
setMigrationProgress({
|
|
164
184
|
id: migration.id,
|
|
165
185
|
rowsDone: 0,
|
|
166
186
|
rowsRemaining: 0,
|
|
167
187
|
state: "failed",
|
|
168
|
-
reason
|
|
188
|
+
reason,
|
|
169
189
|
});
|
|
170
190
|
continue;
|
|
171
191
|
}
|
|
@@ -176,6 +196,13 @@ async function runCycleLocked(deps, lock) {
|
|
|
176
196
|
candidates.push(migration);
|
|
177
197
|
}
|
|
178
198
|
if (candidates.length === 0) {
|
|
199
|
+
// flair#812: mark the cycle done even on the no-op path. "Nothing was
|
|
200
|
+
// pending" and "the cycle never ran" are opposite conclusions that used
|
|
201
|
+
// to produce an IDENTICAL `/HealthDetail` reading (`cyclePhase: "idle",
|
|
202
|
+
// lastCycleAt: null`), which is exactly why a totally-skipped boot cycle
|
|
203
|
+
// looked healthy for months. A cycle that reached this line has done its
|
|
204
|
+
// job, and now says so.
|
|
205
|
+
setCyclePhase("done");
|
|
179
206
|
return { ran: false, reason: "nothing pending" };
|
|
180
207
|
}
|
|
181
208
|
// ── ONE shared async pre-hash at cycle start (K&S: computed after ready,
|
|
@@ -30,11 +30,22 @@ export function readMigrationState(path) {
|
|
|
30
30
|
if (!existsSync(path))
|
|
31
31
|
return {};
|
|
32
32
|
const raw = JSON.parse(readFileSync(path, "utf-8"));
|
|
33
|
-
|
|
33
|
+
if (raw && typeof raw === "object")
|
|
34
|
+
return raw;
|
|
35
|
+
console.warn(`[flair-migrations] ${path} is not a JSON object — ignoring it and re-detecting every migration this boot`);
|
|
36
|
+
return {};
|
|
34
37
|
}
|
|
35
|
-
catch {
|
|
38
|
+
catch (err) {
|
|
36
39
|
// Corrupt/unreadable state — treat as "nothing known yet", never throw.
|
|
37
40
|
// Worst case this costs one extra (cheap, bounded) detect() call.
|
|
41
|
+
//
|
|
42
|
+
// flair#812: but SAY SO. This file is hand-edited in practice (the
|
|
43
|
+
// documented remediation for a stuck migration), and a fat-fingered
|
|
44
|
+
// edit silently discarding every recorded outcome is precisely the
|
|
45
|
+
// class of quiet failure this issue is about. The behaviour is
|
|
46
|
+
// unchanged and still safe — only the silence is fixed.
|
|
47
|
+
console.warn(`[flair-migrations] could not read ${path} (${err?.message ?? String(err)}) — ` +
|
|
48
|
+
`treating migration state as empty and re-detecting every migration this boot`);
|
|
38
49
|
return {};
|
|
39
50
|
}
|
|
40
51
|
}
|
|
@@ -1,25 +1,34 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* models-dir.ts — the ONE place that decides where model GGUFs live.
|
|
3
3
|
*
|
|
4
|
-
* Extracted from embeddings-provider.ts (flair#815) so
|
|
5
|
-
*
|
|
4
|
+
* Extracted from embeddings-provider.ts (flair#815) so a second model
|
|
5
|
+
* consumer could share the exact same resolution WITHOUT importing
|
|
6
6
|
* embeddings-provider: several unit-isolated tests `mock.module()` the whole
|
|
7
7
|
* embeddings-provider module (with only the named exports THEY consume), and
|
|
8
8
|
* bun's module cache is process-wide — so any new cross-module import of
|
|
9
9
|
* embeddings-provider from another resource makes those mocks incomplete and
|
|
10
10
|
* kills unrelated test files at module load. This module is tiny, pure
|
|
11
|
-
* node-stdlib, and never mocked, so
|
|
12
|
-
*
|
|
11
|
+
* node-stdlib, and never mocked, so any model consumer can depend on it
|
|
12
|
+
* safely.
|
|
13
|
+
*
|
|
14
|
+
* The second consumer that motivated the split (the cross-encoder reranker)
|
|
15
|
+
* has since been removed (flair#893), leaving the embedding engine as the
|
|
16
|
+
* only caller. The module stays split anyway, and deliberately: the
|
|
17
|
+
* `mock.module()` hazard above is a property of embeddings-provider, not of
|
|
18
|
+
* the reranker, so folding this back in would re-arm it for whatever the next
|
|
19
|
+
* model consumer turns out to be. It is also the single documented source of
|
|
20
|
+
* truth for the models dir, independently tested as such.
|
|
13
21
|
*/
|
|
14
22
|
import { existsSync } from "node:fs";
|
|
15
23
|
import { homedir } from "node:os";
|
|
16
24
|
import { join } from "node:path";
|
|
17
25
|
/**
|
|
18
|
-
* Resolve the directory model GGUFs live in / download into —
|
|
19
|
-
* embedding engine (embeddings-boot.ts's `register()` options) and the
|
|
20
|
-
*
|
|
21
|
-
*
|
|
22
|
-
* models location
|
|
26
|
+
* Resolve the directory model GGUFs live in / download into — used by the
|
|
27
|
+
* embedding engine (embeddings-boot.ts's `register()` options), and the
|
|
28
|
+
* resolution any future model consumer should use rather than re-deriving its
|
|
29
|
+
* own. flair#815 exists because a consumer that hardcoded `<cwd>/models`
|
|
30
|
+
* silently failed open wherever Harper's cwd wasn't the models location; the
|
|
31
|
+
* fix was to give the decision exactly one home, which is this function.
|
|
23
32
|
*
|
|
24
33
|
* Resolution order (everything writable, never the read-only package dir):
|
|
25
34
|
* 1. FLAIR_MODELS_DIR — explicit operator/docker override.
|
|
@@ -2,7 +2,12 @@ function presenceDelegationContext(auth) {
|
|
|
2
2
|
const agentAuth = auth.kind === "agent"
|
|
3
3
|
? { agentId: auth.agentId, isAdmin: auth.isAdmin }
|
|
4
4
|
: { agentId: "internal", isAdmin: true };
|
|
5
|
-
|
|
5
|
+
// `__flairInternal` marks the `internal` branch as DELIBERATE (flair#936) —
|
|
6
|
+
// an "internal" verdict relayed here is a trusted in-process call this file
|
|
7
|
+
// has already established, not a caller who forgot to pass a context. It is
|
|
8
|
+
// read only by resources/agent-auth.ts's accidental-omission warning; it
|
|
9
|
+
// grants nothing and is never consulted for an authorization decision.
|
|
10
|
+
return { request: { _flairAgentAuth: agentAuth }, __flairInternal: true };
|
|
6
11
|
}
|
|
7
12
|
/**
|
|
8
13
|
* The full presence roster (one row per agent — bounded, per the K&S
|
|
@@ -0,0 +1,149 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* record-owner-guard.ts — ONE enforcement point for "you may not modify a
|
|
3
|
+
* record you do not own".
|
|
4
|
+
*
|
|
5
|
+
* ─── The defect this deletes ─────────────────────────────────────────────────
|
|
6
|
+
*
|
|
7
|
+
* Harper's REST layer maps verbs to resource methods ONE-TO-ONE (see
|
|
8
|
+
* harper/dist/server/REST.js's method switch): GET→get, POST→post, PUT→put,
|
|
9
|
+
* PATCH→patch, DELETE→delete. There is no fallback — `patch()` does NOT route
|
|
10
|
+
* through `put()`. So an ownership rule written inside a resource's `put()` is
|
|
11
|
+
* enforced on PUT and on nothing else, and a resource with no `patch()` override
|
|
12
|
+
* reaches the table with only its `allow*` gate, which is typically
|
|
13
|
+
* `allowVerified()` — "any verified agent".
|
|
14
|
+
*
|
|
15
|
+
* Almost every flair resource wrote its per-record rules in `put()`. That is not
|
|
16
|
+
* a mistake anyone made once; it is what the resource model invites, because
|
|
17
|
+
* `put()` is where the write logic naturally goes and nothing anywhere says the
|
|
18
|
+
* other verbs exist. Fixing it resource-by-resource would mean N hand-written
|
|
19
|
+
* guards — N chances to get one subtly wrong — and would still leave the NEXT
|
|
20
|
+
* resource broken by default, because its author would have to know this file
|
|
21
|
+
* exists to be safe. So the rule lives here instead, once, on the path every
|
|
22
|
+
* HTTP request already takes.
|
|
23
|
+
*
|
|
24
|
+
* ─── The rule, and why it is this narrow ─────────────────────────────────────
|
|
25
|
+
*
|
|
26
|
+
* On a record that ALREADY EXISTS, a non-admin caller whose agent id does not
|
|
27
|
+
* match the record's stored owner is refused — on every mutating verb.
|
|
28
|
+
*
|
|
29
|
+
* It deliberately says nothing about creation. Over-blocking is the failure mode
|
|
30
|
+
* a security fix reaches for, and a blunter rule ("the caller must own whatever
|
|
31
|
+
* it names") would break real flows: Presence heartbeats arrive at the
|
|
32
|
+
* collection with no record yet, credential provisioning creates rows for other
|
|
33
|
+
* principals, and a MemoryGrant's `granteeId` is SUPPOSED to be someone else.
|
|
34
|
+
* Restricting this to records that exist means the guard can only ever narrow
|
|
35
|
+
* mutation of another agent's data. It cannot break a create, and it cannot
|
|
36
|
+
* break an agent writing its own record.
|
|
37
|
+
*
|
|
38
|
+
* ─── Owner is read from STORED STATE, never from the request body ────────────
|
|
39
|
+
*
|
|
40
|
+
* The guards this replaces compared the owner field in the request BODY — the
|
|
41
|
+
* owner the CALLER CLAIMS — and denied only when that field was present and
|
|
42
|
+
* mismatched. A body that simply omitted it was compared against nothing and
|
|
43
|
+
* passed, whatever record the URL named. Harper binds the write to the URL's id,
|
|
44
|
+
* not to the body, so body-derived authorization was answering a question nobody
|
|
45
|
+
* asked. That is the same defect as the verb gap wearing different clothes: an
|
|
46
|
+
* authorization decision made against attacker-supplied data instead of stored
|
|
47
|
+
* state. Both are closed here, together, because closing one leaves the other
|
|
48
|
+
* reachable.
|
|
49
|
+
*
|
|
50
|
+
* Per-resource body checks (no-forge attribution on CREATE) still belong in the
|
|
51
|
+
* resources — they answer a different question, "may you attribute a NEW record
|
|
52
|
+
* to someone else", which this guard does not address.
|
|
53
|
+
*
|
|
54
|
+
* ─── Keeping this honest as the codebase grows ───────────────────────────────
|
|
55
|
+
*
|
|
56
|
+
* OWNER_FIELDS below is static and PR-reviewed, matching the posture of
|
|
57
|
+
* resources/record-types.ts. A static map alone would rot, so
|
|
58
|
+
* test/unit/record-owner-guard-coverage.test.ts parses `schemas/*.graphql` and
|
|
59
|
+
* FAILS when a table declares an owner-shaped column and is neither listed here
|
|
60
|
+
* nor exempted with a stated reason. A table added later enters that test's
|
|
61
|
+
* scope the moment it declares the column — nobody has to know this file exists.
|
|
62
|
+
*/
|
|
63
|
+
/**
|
|
64
|
+
* Table → the attribute naming the principal that owns a row.
|
|
65
|
+
*
|
|
66
|
+
* Derived from the columns actually declared in `schemas/*.graphql`, and pinned
|
|
67
|
+
* against them by the coverage test. Tables with no resource class are listed
|
|
68
|
+
* anyway: costing nothing when the route does not exist is much better than
|
|
69
|
+
* being absent on the day someone adds one.
|
|
70
|
+
*/
|
|
71
|
+
export const OWNER_FIELDS = Object.freeze({
|
|
72
|
+
Credential: "principalId",
|
|
73
|
+
Integration: "agentId",
|
|
74
|
+
Memory: "agentId",
|
|
75
|
+
MemoryCandidate: "agentId",
|
|
76
|
+
MemoryGrant: "ownerId",
|
|
77
|
+
MemoryUsage: "agentId",
|
|
78
|
+
OAuthAuthCode: "principalId",
|
|
79
|
+
OAuthToken: "principalId",
|
|
80
|
+
OrgEvent: "authorId",
|
|
81
|
+
Presence: "agentId",
|
|
82
|
+
Relationship: "agentId",
|
|
83
|
+
Soul: "agentId",
|
|
84
|
+
WorkspaceState: "agentId",
|
|
85
|
+
});
|
|
86
|
+
/**
|
|
87
|
+
* Tables that declare an owner-shaped column but are deliberately NOT guarded
|
|
88
|
+
* here, each with the reason. The coverage test accepts these and rejects
|
|
89
|
+
* anything else, so an omission has to be argued rather than merely happen.
|
|
90
|
+
*/
|
|
91
|
+
export const OWNER_GUARD_EXEMPT = Object.freeze({
|
|
92
|
+
// The principal table's own rule is "the record IS the caller", not "the
|
|
93
|
+
// record has an owner column" — a principal may edit itself, and only an
|
|
94
|
+
// admin may change anyone's admin status. That is enforced in
|
|
95
|
+
// resources/Agent.ts's shared write-authorization helper, which both its
|
|
96
|
+
// put() and its patch() route through.
|
|
97
|
+
Agent: "self-ownership by primary key; enforced in resources/Agent.ts for every verb",
|
|
98
|
+
});
|
|
99
|
+
/** The verbs that can mutate a record, and therefore need the rule applied. */
|
|
100
|
+
export const MUTATING_METHODS = Object.freeze(["POST", "PUT", "PATCH", "DELETE"]);
|
|
101
|
+
export function isMutatingMethod(method) {
|
|
102
|
+
return MUTATING_METHODS.includes(method.toUpperCase());
|
|
103
|
+
}
|
|
104
|
+
/**
|
|
105
|
+
* Resolve a request path to the guarded table and record id it addresses, or
|
|
106
|
+
* null when the path is not a guarded single-record route.
|
|
107
|
+
*
|
|
108
|
+
* Matches `/<Table>/<id>` ONLY. A collection path (`/<Table>`) addresses no
|
|
109
|
+
* existing record, so there is nothing to own and nothing to check — that is the
|
|
110
|
+
* "creation is untouched" property, expressed as a parse rather than a special
|
|
111
|
+
* case. The table segment is matched EXACTLY so `/SoulFeed/x` can never be read
|
|
112
|
+
* as a `Soul` route.
|
|
113
|
+
*/
|
|
114
|
+
export function resolveGuardedRecord(pathname) {
|
|
115
|
+
const parts = pathname.split("/").filter(Boolean);
|
|
116
|
+
if (parts.length < 2)
|
|
117
|
+
return null;
|
|
118
|
+
const table = parts[0];
|
|
119
|
+
const ownerField = OWNER_FIELDS[table];
|
|
120
|
+
if (!ownerField)
|
|
121
|
+
return null;
|
|
122
|
+
let id;
|
|
123
|
+
try {
|
|
124
|
+
id = decodeURIComponent(parts[1]);
|
|
125
|
+
}
|
|
126
|
+
catch {
|
|
127
|
+
return null; // malformed percent-encoding addresses no record we can resolve
|
|
128
|
+
}
|
|
129
|
+
if (!id)
|
|
130
|
+
return null;
|
|
131
|
+
return { table, ownerField, id };
|
|
132
|
+
}
|
|
133
|
+
/**
|
|
134
|
+
* Decide whether a caller may mutate an already-stored record.
|
|
135
|
+
*
|
|
136
|
+
* Pure, so the decision is testable without a Harper instance — the middleware
|
|
137
|
+
* supplies the record it loaded. A record that does not exist, or that carries
|
|
138
|
+
* no owner value, is NOT refused here: the first is a create (or a 404 the
|
|
139
|
+
* resource will produce), and the second is a row with nothing to own, neither
|
|
140
|
+
* of which this rule is about.
|
|
141
|
+
*/
|
|
142
|
+
export function isForbiddenOwnerMutation(record, ownerField, callerAgentId) {
|
|
143
|
+
if (!record)
|
|
144
|
+
return false;
|
|
145
|
+
const owner = record[ownerField];
|
|
146
|
+
if (owner == null || owner === "")
|
|
147
|
+
return false;
|
|
148
|
+
return owner !== callerAgentId;
|
|
149
|
+
}
|
|
@@ -3,8 +3,8 @@
|
|
|
3
3
|
// Extracted from resources/SemanticSearch.ts's post() (Kern-approved
|
|
4
4
|
// refactor, flair#695). Before this module existed,
|
|
5
5
|
// SemanticSearch.post() was one function entangling auth resolution,
|
|
6
|
-
// rate-limiting, HNSW/BM25 retrieval, post-retrieval filtering,
|
|
7
|
-
//
|
|
6
|
+
// rate-limiting, HNSW/BM25 retrieval, post-retrieval filtering, AND
|
|
7
|
+
// retrievalCount/lastRetrieved hit-tracking side
|
|
8
8
|
// effects — so the ONLY way for MemoryBootstrap (resources/MemoryBootstrap.ts)
|
|
9
9
|
// to get bounded, HNSW-pushed-down candidates was to duplicate the retrieval
|
|
10
10
|
// logic or trip the side effects (an internal bootstrap call spuriously
|
|
@@ -16,7 +16,7 @@
|
|
|
16
16
|
// the HNSW leg query construction (sort/select/conditions/limit), the BM25 +
|
|
17
17
|
// union-RRF hybrid fusion, the per-record temporal/expiry/supersede filters,
|
|
18
18
|
// and the scope.isAllowed() defense-in-depth re-check. It does NOT own: auth
|
|
19
|
-
// resolution, rate-limiting,
|
|
19
|
+
// resolution, rate-limiting, or the retrievalCount/lastRetrieved
|
|
20
20
|
// hit-tracking side effects — those stay in SemanticSearch.post()'s wrapper
|
|
21
21
|
// (resources/SemanticSearch.ts) so an internal caller (bootstrap) never trips
|
|
22
22
|
// them.
|
|
@@ -32,7 +32,8 @@
|
|
|
32
32
|
// Returns results AFTER all filters, sorted best-first by `_score` — bounded
|
|
33
33
|
// ONLY by the `limit` the caller chose to push down (the core never
|
|
34
34
|
// multiplies `limit` internally; any overfetch policy — SemanticSearch's
|
|
35
|
-
// CANDIDATE_MULTIPLIER,
|
|
35
|
+
// CANDIDATE_MULTIPLIER, MemoryBootstrap's K formula — is the CALLER's
|
|
36
|
+
// decision, made
|
|
36
37
|
// before calling in). Never exposes which internal leg (BM25+RRF hybrid vs.
|
|
37
38
|
// legacy HNSW-only vs. keyword-only fallback) produced a given result — the
|
|
38
39
|
// output shape is identical regardless of `hybrid`.
|