@panaversity/ksor 0.0.6 → 0.0.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +273 -0
- package/dist/cli.mjs +3573 -1354
- package/dist/index.d.mts +1 -1
- package/dist/index.mjs +1 -1
- package/dist/{src-CpDIVudJ.mjs → src-pl4aOpVs.mjs} +1 -0
- package/docs/index.md +8 -4
- package/package.json +2 -2
- package/schema/migrations/2.1-2.2__governance-on-the-node-row.sql +55 -0
- package/schema/migrations/2.2-2.3__takedown-writes-and-a-readable-ledger.sql +70 -0
- package/schema/migrations/2.3-2.4__a-generation-remembers-its-schema.sql +22 -0
- package/schema/schema.sql +61 -8
- package/templates/scaffold/.agents/skills/format-checker/check.mjs +24 -2
- package/templates/scaffold/.claude/skills/format-checker/check.mjs +24 -2
- package/templates/scaffold/AGENTS.md +103 -9
- package/templates/scaffold/README.md +47 -9
- package/templates/scaffold/env.example +82 -2
- package/templates/scaffold/gitignore +4 -0
- package/templates/scaffold/instance.md +14 -0
- package/templates/scaffold/package.json +7 -3
- package/templates/scaffold/system/site/app/.well-known/mcp/server.json/route.ts +45 -0
- package/templates/scaffold/system/site/lib/audience-rule.ts +44 -0
- package/templates/scaffold/system/site/lib/audience.ts +4 -14
- package/templates/scaffold/system/site/lib/denial-rule.ts +212 -0
- package/templates/scaffold/system/site/lib/shared.ts +48 -0
- package/templates/scaffold/system/site/lib/stage-knowledge.ts +186 -8
package/dist/index.d.mts
CHANGED
|
@@ -21,7 +21,7 @@ declare const exitCodes: {
|
|
|
21
21
|
type ExitCode = (typeof exitCodes)[keyof typeof exitCodes];
|
|
22
22
|
/** The CLI vocabulary. Lifecycle verbs plus the corpus operations the bundled
|
|
23
23
|
* kernel provides (one binary — decision 12 publish revision). */
|
|
24
|
-
declare const verbs: readonly ["init", "dev", "build", "serve", "ingest", "schema", "grant", "calibrate", "gc"];
|
|
24
|
+
declare const verbs: readonly ["init", "dev", "build", "serve", "ingest", "schema", "grant", "takedown", "calibrate", "gc"];
|
|
25
25
|
type Verb = (typeof verbs)[number];
|
|
26
26
|
interface ResolvedCommand {
|
|
27
27
|
/** The first non-flag token, or null when only flags (or nothing) appear. */
|
package/dist/index.mjs
CHANGED
|
@@ -1,2 +1,2 @@
|
|
|
1
|
-
import { n as resolveCommand, r as verbs, t as exitCodes } from "./src-
|
|
1
|
+
import { n as resolveCommand, r as verbs, t as exitCodes } from "./src-pl4aOpVs.mjs";
|
|
2
2
|
export { exitCodes, resolveCommand, verbs };
|
package/docs/index.md
CHANGED
|
@@ -24,10 +24,14 @@ instead of their training memory. The corpus grows with each implemented verb.
|
|
|
24
24
|
at `http://localhost:3000`; `pnpm build` writes a fully static export to
|
|
25
25
|
`system/site/out/`. `KSOR_BASE_PATH=/repo pnpm build` targets sub-path
|
|
26
26
|
hosting.
|
|
27
|
-
- `ksor serve` runs the MCP server over a built record
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
27
|
+
- `ksor serve` runs the MCP server over a built record — the climbed rung,
|
|
28
|
+
needing Postgres and a provider key — alongside the write plane that keeps
|
|
29
|
+
the record current: `ksor schema` (provision or migrate the database),
|
|
30
|
+
`ksor grant` (authorize a tenant for ingest), `ksor ingest` (build and publish
|
|
31
|
+
a generation), `ksor takedown` (withdraw a document from EVERY surface, and
|
|
32
|
+
export the manifest the site build reads), `ksor calibrate` (measure the
|
|
33
|
+
abstention floor) and `ksor gc` (reap retired generations). Only `ksor dev` and `ksor build` remain designed, not
|
|
34
|
+
implemented: each prints an honest notice and exits `2`.
|
|
31
35
|
- Exit codes are a contract: `1` refused (first stderr line is a stable
|
|
32
36
|
slug such as `error: bad-name`, followed by a remedy), `2` designed but
|
|
33
37
|
not implemented, `3` the environment cannot run ksor
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@panaversity/ksor",
|
|
3
|
-
"version": "0.0.
|
|
4
|
-
"description": "Knowledge System of Record —
|
|
3
|
+
"version": "0.0.8",
|
|
4
|
+
"description": "Knowledge System of Record — compile governed markdown into a static site for people and an MCP server for AI agents, with citations and measured abstention.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"abstention",
|
|
7
7
|
"citations",
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
-- 2.1 -> 2.2 · Governance moves onto the node row.
|
|
2
|
+
--
|
|
3
|
+
-- Until now the ingest adapter kept four things from a document's frontmatter
|
|
4
|
+
-- (title, order, an optional sor_id, and the path-derived slug) and discarded
|
|
5
|
+
-- the rest, so `visibility`, the authored `status`, `owner` and `provenance`
|
|
6
|
+
-- existed only in markdown and in whichever surface happened to re-derive them.
|
|
7
|
+
-- The site enforced `visibility:`; the MCP door could not, because the record
|
|
8
|
+
-- did not carry it. This migration gives the record the columns every surface
|
|
9
|
+
-- must read, so the guarantee lives in one place instead of being re-implemented
|
|
10
|
+
-- per surface.
|
|
11
|
+
--
|
|
12
|
+
-- Every column is additive and nullable, so a 2.1 reader still reads a 2.2
|
|
13
|
+
-- database (compatible_from stays 2.0). `corpus_id` is backfilled from the
|
|
14
|
+
-- corpora pointer, which is exact while one corpus serves one tenant — the
|
|
15
|
+
-- state this migration runs in.
|
|
16
|
+
|
|
17
|
+
ALTER TABLE content_nodes
|
|
18
|
+
-- Which record this node belongs to. Content rows were scoped by
|
|
19
|
+
-- (tenant_id, generation) only, while corpora/ingestion_runs/takedown are
|
|
20
|
+
-- keyed (tenant_id, corpus_id); carrying it here closes that split before a
|
|
21
|
+
-- second record exists.
|
|
22
|
+
ADD COLUMN IF NOT EXISTS corpus_id TEXT,
|
|
23
|
+
-- The audience tier the document declares. NULL = the instance's
|
|
24
|
+
-- default_visibility. Enforced at the serving door, not only at site build.
|
|
25
|
+
ADD COLUMN IF NOT EXISTS visibility TEXT,
|
|
26
|
+
-- The AUTHORED governance status (draft / approved / superseded). Distinct
|
|
27
|
+
-- from `status`, which is the SERVING state of the row (published /
|
|
28
|
+
-- draft / archived) and is set by the pipeline, not by the author.
|
|
29
|
+
ADD COLUMN IF NOT EXISTS doc_status TEXT,
|
|
30
|
+
ADD COLUMN IF NOT EXISTS owner TEXT,
|
|
31
|
+
-- Where the document's claims come from, as authored. JSONB so a list of
|
|
32
|
+
-- sources survives without inventing a side table.
|
|
33
|
+
ADD COLUMN IF NOT EXISTS provenance JSONB,
|
|
34
|
+
-- stable_id of the document that replaces this one, when it is superseded.
|
|
35
|
+
ADD COLUMN IF NOT EXISTS superseded_by TEXT;
|
|
36
|
+
|
|
37
|
+
UPDATE content_nodes n
|
|
38
|
+
SET corpus_id = c.corpus_id
|
|
39
|
+
FROM corpora c
|
|
40
|
+
WHERE c.tenant_id = n.tenant_id
|
|
41
|
+
AND n.corpus_id IS NULL;
|
|
42
|
+
|
|
43
|
+
-- NOTE (2.4): this index is DROPPED again by 2.2 -> 2.3, and schema.sql builds
|
|
44
|
+
-- a fresh database without it. The rationale below is wrong and is kept only
|
|
45
|
+
-- because a migration that has run somewhere must not be rewritten: the serving
|
|
46
|
+
-- predicate filters on `coalesce(visibility, <a per-transaction GUC>)`, which no
|
|
47
|
+
-- plain btree on `visibility` can serve, so the index was built and maintained
|
|
48
|
+
-- and never read — the same defect the HNSW arm was fixed for. An operator
|
|
49
|
+
-- reading this file sees the claim, so it is corrected here rather than
|
|
50
|
+
-- silently (round-9 review of PR 43).
|
|
51
|
+
--
|
|
52
|
+
-- The serving filter is (tenant, generation, visibility); without this the
|
|
53
|
+
-- audience predicate turns every search into a scan of the generation.
|
|
54
|
+
CREATE INDEX IF NOT EXISTS idx_nodes_visibility
|
|
55
|
+
ON content_nodes (tenant_id, generation, visibility);
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
-- 2.2 -> 2.3 · A door for takedown, and a ledger someone can read.
|
|
2
|
+
--
|
|
3
|
+
-- Two governance mechanisms were complete except for the permission that makes
|
|
4
|
+
-- them usable:
|
|
5
|
+
--
|
|
6
|
+
-- takedown_denylist the serving-side denial worked perfectly, but ingest
|
|
7
|
+
-- held only SELECT — so the only way to impose a takedown
|
|
8
|
+
-- was a superuser psql prompt, and nothing recorded who
|
|
9
|
+
-- did it.
|
|
10
|
+
-- retrieval_log FORCE row-level security, an INSERT policy, and NO
|
|
11
|
+
-- select policy and no SELECT grant to any role. The
|
|
12
|
+
-- provenance ledger the governance story rests on could
|
|
13
|
+
-- be written and never read. CI only appeared to prove
|
|
14
|
+
-- otherwise because its DSN is a superuser.
|
|
15
|
+
|
|
16
|
+
-- ── takedown: the write plane ────────────────────────────────────────────────
|
|
17
|
+
GRANT INSERT, UPDATE, DELETE ON takedown_denylist TO sor_content_ingest;
|
|
18
|
+
|
|
19
|
+
-- Same shape as ingest_write on the content tables: the tenant GUC must match
|
|
20
|
+
-- AND the grant table must authorize this role for this tenant. A takedown is
|
|
21
|
+
-- a write to the record's governance and is authorized the same way every
|
|
22
|
+
-- other write is.
|
|
23
|
+
DROP POLICY IF EXISTS takedown_write ON takedown_denylist;
|
|
24
|
+
CREATE POLICY takedown_write ON takedown_denylist FOR ALL TO sor_content_ingest
|
|
25
|
+
USING (tenant_id = current_setting('app.tenant_id', true)
|
|
26
|
+
AND EXISTS (SELECT 1 FROM ingest_tenant_grants g
|
|
27
|
+
WHERE g.role_name = current_user
|
|
28
|
+
AND g.tenant_id = takedown_denylist.tenant_id))
|
|
29
|
+
WITH CHECK (tenant_id = current_setting('app.tenant_id', true)
|
|
30
|
+
AND EXISTS (SELECT 1 FROM ingest_tenant_grants g
|
|
31
|
+
WHERE g.role_name = current_user
|
|
32
|
+
AND g.tenant_id = takedown_denylist.tenant_id));
|
|
33
|
+
|
|
34
|
+
-- ── retrieval_log: a role that can actually read the ledger ──────────────────
|
|
35
|
+
DO $$
|
|
36
|
+
BEGIN
|
|
37
|
+
IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'sor_content_auditor') THEN
|
|
38
|
+
CREATE ROLE sor_content_auditor NOLOGIN;
|
|
39
|
+
END IF;
|
|
40
|
+
END $$;
|
|
41
|
+
|
|
42
|
+
GRANT USAGE ON SCHEMA public TO sor_content_auditor;
|
|
43
|
+
GRANT SELECT ON retrieval_log TO sor_content_auditor;
|
|
44
|
+
-- An auditor reads the ledger and the denial list; it can reach no content.
|
|
45
|
+
GRANT SELECT ON takedown_denylist, schema_meta, corpora, ingestion_runs TO sor_content_auditor;
|
|
46
|
+
|
|
47
|
+
-- The tenant wall applies to the auditor exactly as it does to everyone else:
|
|
48
|
+
-- reading the ledger is still scoped to one tenant's rows.
|
|
49
|
+
DROP POLICY IF EXISTS tenant_read ON retrieval_log;
|
|
50
|
+
CREATE POLICY tenant_read ON retrieval_log FOR SELECT
|
|
51
|
+
USING (tenant_id = current_setting('app.tenant_id', true));
|
|
52
|
+
|
|
53
|
+
-- The applying user can assume it, the way it can already assume the other two.
|
|
54
|
+
DO $$
|
|
55
|
+
BEGIN
|
|
56
|
+
EXECUTE format('GRANT sor_content_auditor TO %I WITH SET TRUE', current_user);
|
|
57
|
+
END $$;
|
|
58
|
+
|
|
59
|
+
-- Lifting a denial must be distinguishable from imposing one by the INDEXED
|
|
60
|
+
-- action column, not only by reading each row's JSON detail.
|
|
61
|
+
ALTER TABLE retrieval_log DROP CONSTRAINT IF EXISTS retrieval_log_action_check;
|
|
62
|
+
ALTER TABLE retrieval_log ADD CONSTRAINT retrieval_log_action_check CHECK (action = ANY (ARRAY[
|
|
63
|
+
'content_served','similarity_searched','corpus_seeded','outline_served',
|
|
64
|
+
'search_abstained','generation_activated','takedown_applied','takedown_revoked']));
|
|
65
|
+
|
|
66
|
+
-- Drop an index the serving predicate cannot use: it filters on
|
|
67
|
+
-- `coalesce(visibility, <runtime GUC>)`, which no plain btree on `visibility`
|
|
68
|
+
-- can satisfy. Built and maintained, never read — the same shape as the HNSW
|
|
69
|
+
-- index this release also stopped paying for.
|
|
70
|
+
DROP INDEX IF EXISTS idx_nodes_visibility;
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
-- 2.3 -> 2.4 · A generation records the schema it was built against.
|
|
2
|
+
--
|
|
3
|
+
-- The 2.1 -> 2.2 migration added `content_nodes.visibility` and backfilled only
|
|
4
|
+
-- `corpus_id`, because a migration cannot know what a document's frontmatter
|
|
5
|
+
-- says. So on an upgraded database every PRE-EXISTING node has visibility NULL,
|
|
6
|
+
-- and the serving predicate coalesces NULL to `app.default_visibility` — the
|
|
7
|
+
-- WIDEST tier. An adopter who migrated and did not re-ingest served every
|
|
8
|
+
-- `visibility: restricted` document to every public-tier agent, with the schema
|
|
9
|
+
-- gate green, /ready green, and the boot line reporting the audience model as
|
|
10
|
+
-- enforced (round-5 review of #43).
|
|
11
|
+
--
|
|
12
|
+
-- Nothing could detect that, because a generation had no record of when it was
|
|
13
|
+
-- built. Now it does: `ingestion_runs.schema_version` is stamped at ingest, and
|
|
14
|
+
-- NULL means "built before this column existed" — which is exactly the set of
|
|
15
|
+
-- generations whose governance columns cannot be trusted. `serve` refuses to
|
|
16
|
+
-- boot on one when the record declares an audience model.
|
|
17
|
+
|
|
18
|
+
ALTER TABLE ingestion_runs
|
|
19
|
+
ADD COLUMN IF NOT EXISTS schema_version TEXT;
|
|
20
|
+
|
|
21
|
+
COMMENT ON COLUMN ingestion_runs.schema_version IS
|
|
22
|
+
'The schema_meta version in force when this generation was built. NULL means it predates the governance columns, so its visibility values are absent rather than empty.';
|
package/schema/schema.sql
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
-- sor-content schema v2 — the generational corpus store (specs/platform/generations.md §2 is the
|
|
2
2
|
-- design this implements; specs/platform/spec.md §5 the roles; the legacy schema the quarry).
|
|
3
|
-
--
|
|
3
|
+
-- This file provisions a FRESH database at the current version; an EXISTING one moves forward
|
|
4
|
+
-- through schema/migrations/<from>-<to>__<slug>.sql (spec §9: compatibility is a RANGE, recorded
|
|
5
|
+
-- in schema_meta). Both halves are required — the file alone cannot migrate rows an adopter has.
|
|
4
6
|
--
|
|
5
7
|
-- Carried from legacy verbatim where the eval lock demands it: HNSW (m=16, ef_construction=64)
|
|
6
8
|
-- cosine, 'english' generated tsvector, per-tenant one-embedding-model trigger, fail-closed tenant
|
|
@@ -41,6 +43,12 @@ CREATE TABLE ingestion_runs (
|
|
|
41
43
|
started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
42
44
|
heartbeat_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
43
45
|
finished_at TIMESTAMPTZ,
|
|
46
|
+
-- The schema version in force when this generation was BUILT. A generation
|
|
47
|
+
-- carried forward across the 2.1 -> 2.2 migration has NULL visibility on
|
|
48
|
+
-- every node, which the serving predicate reads as the widest tier — so a
|
|
49
|
+
-- record with an audience model must refuse to serve such a generation
|
|
50
|
+
-- rather than quietly publish restricted documents (2.4).
|
|
51
|
+
schema_version TEXT,
|
|
44
52
|
UNIQUE (tenant_id, corpus_id, generation)
|
|
45
53
|
);
|
|
46
54
|
|
|
@@ -59,6 +67,15 @@ CREATE TABLE content_nodes (
|
|
|
59
67
|
position INT NOT NULL DEFAULT 0,
|
|
60
68
|
permalink TEXT, -- CONFIRMED site route (/docs/…), sitemap-verified at publish; NULL = no proven page URL (a group, or an unlisted route) — never a guess
|
|
61
69
|
status TEXT NOT NULL DEFAULT 'published' CHECK (status IN ('published','draft','archived')),
|
|
70
|
+
-- Governance the AUTHOR declares, carried by the record itself (2.2) so every
|
|
71
|
+
-- surface reads one source instead of re-deriving it from markdown. `status`
|
|
72
|
+
-- above is the SERVING state of the row; `doc_status` is what the document says.
|
|
73
|
+
corpus_id TEXT, -- which record this node belongs to
|
|
74
|
+
visibility TEXT, -- audience tier; NULL = instance default_visibility
|
|
75
|
+
doc_status TEXT, -- draft / approved / superseded, as authored
|
|
76
|
+
owner TEXT,
|
|
77
|
+
provenance JSONB, -- where the claims come from, as authored
|
|
78
|
+
superseded_by TEXT, -- stable_id of the replacement
|
|
62
79
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
63
80
|
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
64
81
|
CONSTRAINT nodes_stable_uniq UNIQUE (tenant_id, generation, stable_id),
|
|
@@ -71,6 +88,12 @@ CREATE TABLE content_nodes (
|
|
|
71
88
|
CREATE INDEX idx_nodes_gen ON content_nodes (tenant_id, generation, kind);
|
|
72
89
|
CREATE INDEX idx_nodes_parent ON content_nodes (parent_id);
|
|
73
90
|
CREATE INDEX idx_nodes_keywords ON content_nodes USING gin (keywords);
|
|
91
|
+
-- NO index on visibility. The serving predicate is
|
|
92
|
+
-- `coalesce(n.visibility, <runtime GUC>) = ANY(...)`, and a plain btree cannot
|
|
93
|
+
-- serve a coalesce over a value that is only known per transaction — the index
|
|
94
|
+
-- would be built and maintained and never read, which is exactly the defect
|
|
95
|
+
-- the HNSW arm was just fixed for. The audience filter rides the
|
|
96
|
+
-- (tenant_id, generation, kind) index that every serving arm already uses.
|
|
74
97
|
CREATE UNIQUE INDEX nodes_root_slug_uniq ON content_nodes (tenant_id, generation, slug) WHERE parent_id IS NULL;
|
|
75
98
|
|
|
76
99
|
CREATE TABLE slug_aliases (
|
|
@@ -117,6 +140,11 @@ CREATE TABLE chunks (
|
|
|
117
140
|
embedding VECTOR(1536), -- NULL while pending/failed; dim = the declared space
|
|
118
141
|
embedding_status TEXT NOT NULL DEFAULT 'embedded'
|
|
119
142
|
CHECK (embedding_status IN ('pending','embedded','failed')),
|
|
143
|
+
-- The text-search configuration is RENDERED from instance.md
|
|
144
|
+
-- (retrieval.text_search_config), the way the embedding dimension is. It
|
|
145
|
+
-- is STORED and GENERATED, so it cannot be changed without a re-ingest —
|
|
146
|
+
-- which is exactly why the record declares it rather than inheriting
|
|
147
|
+
-- 'english' from the DDL (audit finding 20).
|
|
120
148
|
search_tsv TSVECTOR GENERATED ALWAYS AS (to_tsvector('english', content)) STORED,
|
|
121
149
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
122
150
|
embedded_at TIMESTAMPTZ,
|
|
@@ -171,7 +199,7 @@ CREATE TABLE retrieval_log (
|
|
|
171
199
|
actor TEXT NOT NULL, -- NO default: unset errors loudly (carried)
|
|
172
200
|
action TEXT NOT NULL CHECK (action IN
|
|
173
201
|
('content_served','similarity_searched','corpus_seeded','outline_served',
|
|
174
|
-
'search_abstained','generation_activated','takedown_applied')),
|
|
202
|
+
'search_abstained','generation_activated','takedown_applied','takedown_revoked')),
|
|
175
203
|
source_id TEXT,
|
|
176
204
|
-- spec §7 audit fields (explicit + queryable; free detail rides JSONB)
|
|
177
205
|
content_hash TEXT,
|
|
@@ -193,9 +221,16 @@ CREATE TABLE schema_meta (
|
|
|
193
221
|
compatible_from TEXT NOT NULL,
|
|
194
222
|
applied_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
|
195
223
|
);
|
|
196
|
-
-- 2.1 adds takedown_denylist.scope (decision 14)
|
|
197
|
-
--
|
|
198
|
-
|
|
224
|
+
-- 2.1 adds takedown_denylist.scope (decision 14). 2.2 puts governance on the node
|
|
225
|
+
-- row (corpus_id, visibility, doc_status, owner, provenance, superseded_by).
|
|
226
|
+
-- 2.3 gives takedown a write plane and the ledger a reader (sor_content_auditor).
|
|
227
|
+
-- 2.4 stamps each generation with the schema it was built against, so a
|
|
228
|
+
-- generation predating the governance columns can be REFUSED rather than served
|
|
229
|
+
-- at default_visibility.
|
|
230
|
+
-- Both are additive and nullable, so a 2.0 reader still reads a 2.2 database —
|
|
231
|
+
-- compatible_from stays 2.0. Existing databases move forward through
|
|
232
|
+
-- schema/migrations/; schema.sql provisions a FRESH one at the current version.
|
|
233
|
+
INSERT INTO schema_meta (schema_version, compatible_from) VALUES ('2.4', '2.0');
|
|
199
234
|
|
|
200
235
|
CREATE OR REPLACE FUNCTION touch_updated_at() RETURNS trigger AS $$
|
|
201
236
|
BEGIN NEW.updated_at = now(); RETURN NEW; END; $$ LANGUAGE plpgsql;
|
|
@@ -229,6 +264,9 @@ CREATE TABLE ingest_tenant_grants (
|
|
|
229
264
|
DO $$ BEGIN
|
|
230
265
|
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'sor_content_runtime') THEN CREATE ROLE sor_content_runtime NOLOGIN; END IF;
|
|
231
266
|
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'sor_content_ingest') THEN CREATE ROLE sor_content_ingest NOLOGIN; END IF;
|
|
267
|
+
-- The ledger's READER (2.3). Without it retrieval_log was write-only under
|
|
268
|
+
-- every credential ksor ships: FORCE RLS, an INSERT policy, and no way back in.
|
|
269
|
+
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'sor_content_auditor') THEN CREATE ROLE sor_content_auditor NOLOGIN; END IF;
|
|
232
270
|
END $$;
|
|
233
271
|
|
|
234
272
|
-- Explicit in-schema membership for the APPLYING role, so SET LOCAL ROLE works from day one
|
|
@@ -236,10 +274,10 @@ END $$;
|
|
|
236
274
|
-- every fresh project until hand-fixed). Deployment LOGIN roles receive the same membership
|
|
237
275
|
-- when they are provisioned (1h).
|
|
238
276
|
DO $$ BEGIN
|
|
239
|
-
EXECUTE format('GRANT sor_content_runtime, sor_content_ingest TO %I WITH SET TRUE', current_user);
|
|
277
|
+
EXECUTE format('GRANT sor_content_runtime, sor_content_ingest, sor_content_auditor TO %I WITH SET TRUE', current_user);
|
|
240
278
|
END $$;
|
|
241
279
|
|
|
242
|
-
GRANT USAGE ON SCHEMA public TO sor_content_runtime, sor_content_ingest;
|
|
280
|
+
GRANT USAGE ON SCHEMA public TO sor_content_runtime, sor_content_ingest, sor_content_auditor;
|
|
243
281
|
-- runtime: read published corpora, write the ledger — NOTHING else.
|
|
244
282
|
GRANT SELECT ON corpora, content_nodes, sources, chunks, slug_aliases, node_centroids, takedown_denylist, schema_meta TO sor_content_runtime;
|
|
245
283
|
-- freshness on /health (2026-07-16): the runtime reads the active run's source_commit +
|
|
@@ -248,7 +286,15 @@ GRANT SELECT ON ingestion_runs TO sor_content_runtime;
|
|
|
248
286
|
GRANT INSERT ON retrieval_log TO sor_content_runtime;
|
|
249
287
|
GRANT USAGE, SELECT ON SEQUENCE retrieval_log_id_seq TO sor_content_runtime;
|
|
250
288
|
-- ingest: build generations + flip, for AUTHORIZED tenants only (policy-checked via the grant table).
|
|
251
|
-
GRANT SELECT ON schema_meta,
|
|
289
|
+
GRANT SELECT ON schema_meta, ingest_tenant_grants TO sor_content_ingest;
|
|
290
|
+
-- Takedown is a WRITE to the record's governance: ingest imposes and lifts it
|
|
291
|
+
-- through `ksor takedown`, authorized by the same grant table every other write
|
|
292
|
+
-- is (2.3 — before it, the only door was a superuser psql prompt).
|
|
293
|
+
GRANT SELECT, INSERT, UPDATE, DELETE ON takedown_denylist TO sor_content_ingest;
|
|
294
|
+
-- The ledger needs a READER. Without one, retrieval_log was write-only under
|
|
295
|
+
-- every credential ksor ships (2.3).
|
|
296
|
+
GRANT SELECT ON retrieval_log TO sor_content_auditor;
|
|
297
|
+
GRANT SELECT ON takedown_denylist, schema_meta, corpora, ingestion_runs TO sor_content_auditor;
|
|
252
298
|
GRANT SELECT, INSERT, UPDATE, DELETE ON corpora, ingestion_runs, content_nodes, sources, chunks, slug_aliases, node_centroids TO sor_content_ingest;
|
|
253
299
|
GRANT INSERT ON retrieval_log TO sor_content_ingest;
|
|
254
300
|
GRANT USAGE, SELECT ON SEQUENCE retrieval_log_id_seq, ingestion_runs_run_id_seq TO sor_content_ingest;
|
|
@@ -274,10 +320,17 @@ CREATE POLICY tenant_read ON node_centroids FOR SELECT USING (tenant_id = cur
|
|
|
274
320
|
CREATE POLICY tenant_read ON takedown_denylist FOR SELECT USING (tenant_id = current_setting('app.tenant_id', true));
|
|
275
321
|
CREATE POLICY tenant_read ON ingestion_runs FOR SELECT USING (tenant_id = current_setting('app.tenant_id', true));
|
|
276
322
|
-- Ledger writes:
|
|
323
|
+
CREATE POLICY tenant_read ON retrieval_log FOR SELECT
|
|
324
|
+
USING (tenant_id = current_setting('app.tenant_id', true));
|
|
277
325
|
CREATE POLICY tenant_write ON retrieval_log FOR INSERT
|
|
278
326
|
WITH CHECK (tenant_id = current_setting('app.tenant_id', true));
|
|
279
327
|
-- Ingest mutations: tenant GUC match AND the grant table authorizes THIS role for THIS tenant
|
|
280
328
|
-- (a CLI flag is not authorization — spec §5).
|
|
329
|
+
CREATE POLICY takedown_write ON takedown_denylist FOR ALL TO sor_content_ingest
|
|
330
|
+
USING (tenant_id = current_setting('app.tenant_id', true)
|
|
331
|
+
AND EXISTS (SELECT 1 FROM ingest_tenant_grants g WHERE g.role_name = current_user AND g.tenant_id = takedown_denylist.tenant_id))
|
|
332
|
+
WITH CHECK (tenant_id = current_setting('app.tenant_id', true)
|
|
333
|
+
AND EXISTS (SELECT 1 FROM ingest_tenant_grants g WHERE g.role_name = current_user AND g.tenant_id = takedown_denylist.tenant_id));
|
|
281
334
|
CREATE POLICY ingest_write ON content_nodes FOR ALL TO sor_content_ingest
|
|
282
335
|
USING (tenant_id = current_setting('app.tenant_id', true)
|
|
283
336
|
AND EXISTS (SELECT 1 FROM ingest_tenant_grants g WHERE g.role_name = current_user AND g.tenant_id = content_nodes.tenant_id))
|
|
@@ -121,6 +121,16 @@ function parseFrontmatter(text) {
|
|
|
121
121
|
for (const raw of match[1].split("\n")) {
|
|
122
122
|
const line = raw.replace(/[ \t]+$/, "");
|
|
123
123
|
if (line === "") continue;
|
|
124
|
+
// A full-line `#` comment is YAML, and the kernel's own instance parser
|
|
125
|
+
// skips it — the checker refusing one made the two grammars disagree about
|
|
126
|
+
// the same file, which is exactly the drift this checker exists to stop.
|
|
127
|
+
// A TRAILING `# ...` is a different question and this is not the place it
|
|
128
|
+
// is decided: `scalarValue` below strips it, exactly as the kernel's
|
|
129
|
+
// frontmatter reader and the site's both do. (An earlier comment here
|
|
130
|
+
// claimed the opposite — that a trailing comment stays part of the value —
|
|
131
|
+
// which contradicted this file's own reader and both build scanners;
|
|
132
|
+
// round-9 review of PR 43.)
|
|
133
|
+
if (line.trimStart().startsWith("#")) continue;
|
|
124
134
|
// YAML requires a space after the colon and refuses tab indentation —
|
|
125
135
|
// both parsed here fine and failed the build (review findings, 2026-08-18).
|
|
126
136
|
if (/^[A-Za-z_][\w-]*:\S/.test(line)) tightColons.push(line);
|
|
@@ -630,8 +640,12 @@ if (!existsSync(knowledgeDir)) {
|
|
|
630
640
|
crossings.push({ kind: "superseded_by", rel, from: p, to: resolved, target: successor });
|
|
631
641
|
}
|
|
632
642
|
}
|
|
633
|
-
// visibility: one audience per document, from the set instance.md declares
|
|
634
|
-
|
|
643
|
+
// visibility: one audience per document, from the set instance.md declares.
|
|
644
|
+
// Through scalarValue, so the comparison sees what the READERS see: a raw
|
|
645
|
+
// read reported `visibility "internal # narrowed 2026-08" is not a
|
|
646
|
+
// declared audience` for a document every reader resolves to `internal`
|
|
647
|
+
// (round-9 review of PR 43).
|
|
648
|
+
const visibility = scalarValue(fm, "visibility");
|
|
635
649
|
const listed = fm.lists.get("visibility") ?? [];
|
|
636
650
|
// A flow list ([a, b]) is already named by the shape rule above.
|
|
637
651
|
const flowList = !fm.quoted.has("visibility") && /^\[.*\]$/.test(visibility ?? "");
|
|
@@ -722,6 +736,14 @@ const INSTANCE_KEYS = new Set([
|
|
|
722
736
|
"site",
|
|
723
737
|
"audiences",
|
|
724
738
|
"default_visibility",
|
|
739
|
+
// Where this record's MCP surface is published. The site emits it as
|
|
740
|
+
// /.well-known/mcp/server.json so an agent can DISCOVER the record rather
|
|
741
|
+
// than being told the URL (AGENTS.md critical rule 3).
|
|
742
|
+
"mcp_url",
|
|
743
|
+
// The record's published semver. The MCP discovery document REQUIRES a
|
|
744
|
+
// version, so this key has to be declarable — and therefore accepted here,
|
|
745
|
+
// or `pnpm check` refuses the very thing the site needs.
|
|
746
|
+
"version",
|
|
725
747
|
"database",
|
|
726
748
|
"embedding",
|
|
727
749
|
"retrieval",
|
|
@@ -121,6 +121,16 @@ function parseFrontmatter(text) {
|
|
|
121
121
|
for (const raw of match[1].split("\n")) {
|
|
122
122
|
const line = raw.replace(/[ \t]+$/, "");
|
|
123
123
|
if (line === "") continue;
|
|
124
|
+
// A full-line `#` comment is YAML, and the kernel's own instance parser
|
|
125
|
+
// skips it — the checker refusing one made the two grammars disagree about
|
|
126
|
+
// the same file, which is exactly the drift this checker exists to stop.
|
|
127
|
+
// A TRAILING `# ...` is a different question and this is not the place it
|
|
128
|
+
// is decided: `scalarValue` below strips it, exactly as the kernel's
|
|
129
|
+
// frontmatter reader and the site's both do. (An earlier comment here
|
|
130
|
+
// claimed the opposite — that a trailing comment stays part of the value —
|
|
131
|
+
// which contradicted this file's own reader and both build scanners;
|
|
132
|
+
// round-9 review of PR 43.)
|
|
133
|
+
if (line.trimStart().startsWith("#")) continue;
|
|
124
134
|
// YAML requires a space after the colon and refuses tab indentation —
|
|
125
135
|
// both parsed here fine and failed the build (review findings, 2026-08-18).
|
|
126
136
|
if (/^[A-Za-z_][\w-]*:\S/.test(line)) tightColons.push(line);
|
|
@@ -630,8 +640,12 @@ if (!existsSync(knowledgeDir)) {
|
|
|
630
640
|
crossings.push({ kind: "superseded_by", rel, from: p, to: resolved, target: successor });
|
|
631
641
|
}
|
|
632
642
|
}
|
|
633
|
-
// visibility: one audience per document, from the set instance.md declares
|
|
634
|
-
|
|
643
|
+
// visibility: one audience per document, from the set instance.md declares.
|
|
644
|
+
// Through scalarValue, so the comparison sees what the READERS see: a raw
|
|
645
|
+
// read reported `visibility "internal # narrowed 2026-08" is not a
|
|
646
|
+
// declared audience` for a document every reader resolves to `internal`
|
|
647
|
+
// (round-9 review of PR 43).
|
|
648
|
+
const visibility = scalarValue(fm, "visibility");
|
|
635
649
|
const listed = fm.lists.get("visibility") ?? [];
|
|
636
650
|
// A flow list ([a, b]) is already named by the shape rule above.
|
|
637
651
|
const flowList = !fm.quoted.has("visibility") && /^\[.*\]$/.test(visibility ?? "");
|
|
@@ -722,6 +736,14 @@ const INSTANCE_KEYS = new Set([
|
|
|
722
736
|
"site",
|
|
723
737
|
"audiences",
|
|
724
738
|
"default_visibility",
|
|
739
|
+
// Where this record's MCP surface is published. The site emits it as
|
|
740
|
+
// /.well-known/mcp/server.json so an agent can DISCOVER the record rather
|
|
741
|
+
// than being told the URL (AGENTS.md critical rule 3).
|
|
742
|
+
"mcp_url",
|
|
743
|
+
// The record's published semver. The MCP discovery document REQUIRES a
|
|
744
|
+
// version, so this key has to be declarable — and therefore accepted here,
|
|
745
|
+
// or `pnpm check` refuses the very thing the site needs.
|
|
746
|
+
"version",
|
|
725
747
|
"database",
|
|
726
748
|
"embedding",
|
|
727
749
|
"retrieval",
|
|
@@ -52,8 +52,9 @@ pnpm check # the format checker — run before handing off any knowledge c
|
|
|
52
52
|
with honest abstention. It is the climbed rung — not required for `pnpm dev`.
|
|
53
53
|
Stand it up in this order (each step's errors explain how to fix themselves):
|
|
54
54
|
|
|
55
|
-
1. **Configure `instance.md`.** One block is required
|
|
56
|
-
|
|
55
|
+
1. **Configure `instance.md`.** One block is required, and it is already
|
|
56
|
+
there, commented out — uncomment it. It names the environment variable
|
|
57
|
+
holding your DSN (never the DSN itself):
|
|
57
58
|
|
|
58
59
|
```yaml
|
|
59
60
|
database:
|
|
@@ -90,12 +91,58 @@ Stand it up in this order (each step's errors explain how to fix themselves):
|
|
|
90
91
|
intended dev shape. A PUBLIC deployment configures the SSO door instead —
|
|
91
92
|
see the comments in `.env.example` and "Serving safely" below.
|
|
92
93
|
|
|
93
|
-
3. **Bring it up
|
|
94
|
+
3. **Bring it up.** Once, then every time:
|
|
94
95
|
|
|
95
96
|
```sh
|
|
96
|
-
pnpm
|
|
97
|
+
pnpm provision # schema (or migrate) + grant — the privileged acts, run once
|
|
98
|
+
pnpm refresh # ingest the record, collect retired generations
|
|
99
|
+
pnpm serve # the MCP server (one supervised process)
|
|
97
100
|
```
|
|
98
101
|
|
|
102
|
+
`provision` is separate on purpose: applying DDL and granting ingest are acts
|
|
103
|
+
an operator performs, not side effects of starting a server. (It is not
|
|
104
|
+
called `setup` because `pnpm setup` is pnpm's own command and would shadow
|
|
105
|
+
it — the step would print "No changes to the environment were made" and do
|
|
106
|
+
nothing.)
|
|
107
|
+
|
|
108
|
+
**Deploying to a container runtime you do not control** (Cloud Run, Fly,
|
|
109
|
+
Container Apps — anything that scales to zero and hands you a `$PORT`):
|
|
110
|
+
`ksor serve` is already shaped for it, and the posture is deliberate.
|
|
111
|
+
|
|
112
|
+
- It binds `$PORT` on `0.0.0.0` when the platform sets one.
|
|
113
|
+
- It holds **no idle database connections**. The pool minimum is 0 and an
|
|
114
|
+
unused connection is closed after 10s, so an idle instance keeps nothing
|
|
115
|
+
open against a serverless Postgres — and a busy one still reuses
|
|
116
|
+
connections instead of paying a TLS handshake per request.
|
|
117
|
+
- The first request after an idle period wakes the database, and that
|
|
118
|
+
connect is **retried** rather than failing: a cold start is a transient,
|
|
119
|
+
not a refusal.
|
|
120
|
+
- `SIGTERM` drains and exits within 8s, inside the ~10s a runtime usually
|
|
121
|
+
allows before `SIGKILL`.
|
|
122
|
+
|
|
123
|
+
What it does NOT do is open and close a connection per request. That is the
|
|
124
|
+
pattern connection poolers exist to remove — the handshake alone is ~26x the
|
|
125
|
+
cost of a pooled query even on localhost, before TLS — and it is not what
|
|
126
|
+
managed Postgres vendors recommend for a process that serves many requests.
|
|
127
|
+
Per-request connections are the right shape only for a per-invocation
|
|
128
|
+
runtime (an edge function), which is a different deployment and would want
|
|
129
|
+
an HTTP database driver rather than TCP.
|
|
130
|
+
|
|
131
|
+
Set `KSOR_SNAPSHOT_KEYS` for any such deployment: without it each cold start
|
|
132
|
+
mints a new signing key, so citations pinned before a scale-down stop
|
|
133
|
+
validating after it.
|
|
134
|
+
|
|
135
|
+
**`pnpm serve` serves; it does not publish.** It is `ksor serve` and nothing
|
|
136
|
+
else — it opens the port against whatever generation is already active and
|
|
137
|
+
needs no ingest privileges. Publishing is `pnpm ingest` (or `pnpm refresh`),
|
|
138
|
+
and it is a separate step ON PURPOSE: a container that re-ingested on boot
|
|
139
|
+
would pay the whole record's embedding cost on every cold start and would
|
|
140
|
+
need write credentials at runtime. So in a deployment, run `pnpm provision` and
|
|
141
|
+
`pnpm ingest` as DEPLOY steps and run `ksor serve` in the container, where
|
|
142
|
+
it honours `$PORT` and binds `0.0.0.0`. If you skip the ingest step, the
|
|
143
|
+
container serves the last generation you flipped — and on a FIRST deploy,
|
|
144
|
+
that is nothing at all.
|
|
145
|
+
|
|
99
146
|
Every step is re-runnable, so this is also how you **refresh after editing
|
|
100
147
|
`knowledge/`**: an applied schema reports "already applied", an existing
|
|
101
148
|
grant reports "already granted", and ingest builds a fresh generation.
|
|
@@ -108,8 +155,14 @@ Stand it up in this order (each step's errors explain how to fix themselves):
|
|
|
108
155
|
unchanged chunks carry forward by content hash, so a rerun on an untouched
|
|
109
156
|
corpus makes **zero provider calls** (`embedded 0, carried N`).
|
|
110
157
|
|
|
111
|
-
|
|
112
|
-
|
|
158
|
+
A rerun on an unchanged record costs **nothing at all**: ingest compares the
|
|
159
|
+
corpus it just read against the generation already serving and, when they
|
|
160
|
+
are identical at the same commit, consumes no generation and writes no rows
|
|
161
|
+
("unchanged — generation N already serves this corpus"). Edit a document and
|
|
162
|
+
the next run builds a generation for it, re-embedding only what changed.
|
|
163
|
+
|
|
164
|
+
Generations do accumulate as you edit. Reap the superseded ones when you
|
|
165
|
+
think of it, or on a schedule:
|
|
113
166
|
|
|
114
167
|
```sh
|
|
115
168
|
pnpm exec ksor gc --instance instance.md
|
|
@@ -162,7 +215,9 @@ abandoned ones.
|
|
|
162
215
|
|
|
163
216
|
### Serving safely (fail-closed posture)
|
|
164
217
|
|
|
165
|
-
`pnpm serve`
|
|
218
|
+
`pnpm serve` **refuses to boot unauthenticated** — there is no auth-off
|
|
219
|
+
default. A local run says so deliberately with `KSOR_AUTH_DISABLED=1` and binds
|
|
220
|
+
loopback, which is the intended dev shape. A **public**
|
|
166
221
|
bind refuses to boot unless auth is configured (`KSOR_SSO_URL` +
|
|
167
222
|
`KSOR_MCP_RESOURCE_URL` + `KSOR_JWT_ALLOWED_AUDIENCES`, making it an OAuth
|
|
168
223
|
Resource Server) OR you deliberately set `KSOR_ALLOW_PUBLIC_UNAUTHENTICATED=1`.
|
|
@@ -179,12 +234,47 @@ Two things worth being deliberate about:
|
|
|
179
234
|
- **Set `KSOR_SSO_ISSUER` when your SSO stamps a stable `iss`.** Audience is
|
|
180
235
|
always enforced against `KSOR_JWT_ALLOWED_AUDIENCES`; naming the issuer adds
|
|
181
236
|
one more check for the cost of one variable.
|
|
237
|
+
- **Set `KSOR_JWKS_URL` unless your SSO is Better Auth.** The signing keys are
|
|
238
|
+
fetched from `<KSOR_SSO_URL>/api/auth/jwks` by default, which is Better
|
|
239
|
+
Auth's layout. Auth0, Okta, Entra, Keycloak and Cognito publish theirs
|
|
240
|
+
elsewhere, and a wrong JWKS URL fails as a transient fetch error — the door
|
|
241
|
+
boots clean and every request 503s with nothing naming the cause.
|
|
242
|
+
|
|
243
|
+
## Withdrawing a document — `ksor takedown`
|
|
244
|
+
|
|
245
|
+
A takedown is the one governance act that must reach EVERY surface at once.
|
|
246
|
+
It needs the database (the denial is a row, not a file), so it belongs to the
|
|
247
|
+
served rung.
|
|
248
|
+
|
|
249
|
+
```sh
|
|
250
|
+
pnpm exec ksor takedown --instance instance.md <stable-id> --reason "legal request 2026-08"
|
|
251
|
+
pnpm exec ksor takedown --instance instance.md <stable-id> --reason "..." --subtree
|
|
252
|
+
pnpm exec ksor takedown --instance instance.md --list # what is currently denied
|
|
253
|
+
pnpm exec ksor takedown --instance instance.md --ledger # who denied what, when
|
|
254
|
+
pnpm exec ksor takedown --instance instance.md --revoke <stable-id>
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
The stable id is what a search result reports as `provenance.stable_id` — for
|
|
258
|
+
most documents that is `knowledge/<path-without-.md>`. `--subtree` withdraws a
|
|
259
|
+
section and everything beneath it, including documents added later.
|
|
260
|
+
`--actor NAME` names who performed the act in the ledger; it defaults to the
|
|
261
|
+
operating user.
|
|
262
|
+
|
|
263
|
+
**The MCP door stops serving it immediately. The SITE stops at its next
|
|
264
|
+
build** — the site reads a file, not the database, and `pnpm build` refreshes
|
|
265
|
+
that file for you (`pnpm export-denylist`). So after a takedown, rebuild and
|
|
266
|
+
redeploy the site, or the human surface keeps publishing what the agent
|
|
267
|
+
surface already refuses.
|
|
182
268
|
|
|
183
269
|
## Publishing
|
|
184
270
|
|
|
185
271
|
`pnpm build` emits a fully static site (`system/site/out/`) deployable to
|
|
186
272
|
any host — Vercel reads the shipped `vercel.json` (deploy from the repo
|
|
187
273
|
ROOT, never `system/site/`), and every other host just serves the folder.
|
|
274
|
+
Once `instance.md` declares a `database:`, `pnpm build` needs `KSOR_DB_URL` as
|
|
275
|
+
well: it runs `pnpm export-denylist` first, which asks the database what has
|
|
276
|
+
been withdrawn and writes `.ksor-denylist.json`. Without the DSN the build
|
|
277
|
+
refuses rather than publish a document someone took down.
|
|
188
278
|
`KSOR_BASE_PATH=/repo pnpm build` targets sub-path hosting. With
|
|
189
279
|
`audiences:` declared, plain `pnpm build` is always the public tier;
|
|
190
280
|
`KSOR_AUDIENCE=<audience> pnpm build` builds a wider tier that belongs
|
|
@@ -211,8 +301,12 @@ Details in README → Deploying.
|
|
|
211
301
|
- `visibility:` names the one audience a document belongs to — a single value
|
|
212
302
|
from `instance.md`'s `audiences:`, never a list, and orthogonal to `status:`
|
|
213
303
|
(an approved document can be restricted, and a draft is not hidden). Leave
|
|
214
|
-
it off and the document takes `default_visibility`.
|
|
215
|
-
|
|
304
|
+
it off and the document takes `default_visibility`. Using the key WITHOUT
|
|
305
|
+
an `audiences:` block is refused on both surfaces — `pnpm build` stops with
|
|
306
|
+
`ksor-visibility-without-audiences` and `pnpm serve` refuses to boot —
|
|
307
|
+
because a document marked restricted while nothing enforces it is the one
|
|
308
|
+
shape where the frontmatter is the only trace of a restriction that is not
|
|
309
|
+
happening. Once `audiences:` is declared, `pnpm check`
|
|
216
310
|
refuses any link or `superseded_by:` pointing from a wider audience at a
|
|
217
311
|
narrower one — the leak no single build can catch, because the build that
|
|
218
312
|
publishes the link has already dropped its target.
|