@panaversity/ksor 0.0.53 → 0.0.54
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +149 -0
- package/README.md +30 -0
- package/dist/cli.mjs +266 -46
- package/dist/{gateway-api-Ck1l_Sgs-B-_wMYLB.mjs → gateway-api-uhx2l1kC-C2BAxISt.mjs} +197 -17
- package/dist/gateway.mjs +1 -1
- package/docs/deploying.md +41 -8
- package/docs/ingesting.md +25 -8
- package/package.json +1 -1
- package/templates/scaffold/AGENTS.md +34 -5
- package/templates/scaffold/README.md +55 -7
- package/templates/scaffold/mcp.json +13 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,154 @@
|
|
|
1
1
|
# @panaversity/ksor
|
|
2
2
|
|
|
3
|
+
## 0.0.54
|
|
4
|
+
|
|
5
|
+
### Patch Changes
|
|
6
|
+
|
|
7
|
+
- 3f55e91: `ksor calibrate` names the free-tier path when a quota refuses it, and the
|
|
8
|
+
calibration text model moves to `gemini-3.7-flash`.
|
|
9
|
+
|
|
10
|
+
Walked on a real free-tier key: embedding is free of charge and a first corpus
|
|
11
|
+
embeds fine (23 chunks, 0 failed), but the DEFAULT calibration door writes one
|
|
12
|
+
probe question per sampled passage with an LLM — and the free tier allows five
|
|
13
|
+
generations a minute. So the documented way to turn on the product's headline
|
|
14
|
+
feature failed, surfacing the vendor's sentence and nothing else.
|
|
15
|
+
|
|
16
|
+
Two quotas reach that code and they need opposite answers: the generation cap is
|
|
17
|
+
a wall no wait clears (the remedy is `--queries-file`, the zero-LLM door), and
|
|
18
|
+
the embedding cap is a per-minute window (the remedy is to wait, and the usual
|
|
19
|
+
cause is an ingest immediately before). Both are now named, with why. A 429 this
|
|
20
|
+
does not recognise is re-thrown untouched — an invented remedy is worse than the
|
|
21
|
+
vendor's own message.
|
|
22
|
+
|
|
23
|
+
`docs/ingesting.md` documents the zero-LLM door where the reader meets the
|
|
24
|
+
command, including how to choose the questions: the floor is set by the weakest
|
|
25
|
+
one, so a vague question drags it down and a question the record cannot answer
|
|
26
|
+
invalidates the measurement.
|
|
27
|
+
|
|
28
|
+
The text model moves `gemini-2.5-flash` → `gemini-3.7-flash`. Cheap, unlike the
|
|
29
|
+
embedding model: it only writes probe questions, so nothing stored is
|
|
30
|
+
re-computed and no floor is invalidated — and the door is recorded beside every
|
|
31
|
+
number, which is what stops two measurements being compared as one experiment.
|
|
32
|
+
|
|
33
|
+
**And calibration now embeds on the patient retry.** It used the READ plane's
|
|
34
|
+
door, which never retries a 429 — correct for a live search, which should
|
|
35
|
+
degrade to keyword-only in under a second rather than stall a reader behind
|
|
36
|
+
backoff, and wrong for a measurement nobody is waiting on. So a free-tier key
|
|
37
|
+
that rate-limited mid-run refused the whole calibration. The intent stays
|
|
38
|
+
`query` (a floor must be measured through the label the door searches with);
|
|
39
|
+
only the retry policy moves, to the one `isRetryable`'s own comment describes
|
|
40
|
+
for batch work. Calibration's text generation already took that path, so this
|
|
41
|
+
was the two halves of one act disagreeing.
|
|
42
|
+
|
|
43
|
+
- dd6371b: Two things: a false claim removed from a shipped page, and the scaffold gains
|
|
44
|
+
`.mcp.json`.
|
|
45
|
+
|
|
46
|
+
**The false claim.** `docs/deploying.md` told adopters "The MCP surface already
|
|
47
|
+
applies the audience scope **per request**", under the heading of the very
|
|
48
|
+
requirement it does not meet. It does not: `content-gateway/src/compose.ts`
|
|
49
|
+
reads `KSOR_AUDIENCE` from the environment once at boot into a per-process
|
|
50
|
+
viewer, and the request path never touches it — `docs/authorization.md` says so
|
|
51
|
+
plainly ("Any caller holding a valid token gets the whole record") and
|
|
52
|
+
`specs/ksor/serve/spec.md` names per-request visibility filtering as out of
|
|
53
|
+
scope. A reader who believed the page would point every caller at one door and
|
|
54
|
+
serve them the restricted half. The page now says what the door does — one
|
|
55
|
+
viewer per door, so one process per audience — and separates the audit it does
|
|
56
|
+
give (a `retrieval_log` row naming the verified caller) from the authorization
|
|
57
|
+
it does not. A docs-truth assertion now fails on the claim itself, not merely on
|
|
58
|
+
a command that no longer exists.
|
|
59
|
+
|
|
60
|
+
**`.mcp.json`.** The scaffold's closed root set gains one member: the MCP
|
|
61
|
+
servers a coding agent may reach from the project. It ships with Neon's, which
|
|
62
|
+
turns the step the tool could never do for an adopter — provision a Postgres,
|
|
63
|
+
enable pgvector, produce a connection string — into four real tool calls
|
|
64
|
+
(`create_project`, `run_sql`, `create_branch`, `get_connection_string`) and one
|
|
65
|
+
sentence to the agent. The scaffold's README and AGENTS.md carry that sentence,
|
|
66
|
+
and both now say plainly which step no agent can do at any price: the embedding
|
|
67
|
+
API key, which no vendor mints over a protocol. Committed rather than ignored,
|
|
68
|
+
because both entries authenticate interactively and the file carries no secret —
|
|
69
|
+
stated, because pasting an API key into it would change that.
|
|
70
|
+
|
|
71
|
+
- eebd777: OpenAI joins Gemini behind the embedding seam, and the wiring stops naming one
|
|
72
|
+
vendor (issue #25).
|
|
73
|
+
|
|
74
|
+
The seam was already vendor-neutral in shape — `EmbeddingProvider`, the
|
|
75
|
+
framework's normalization and degeneracy checks, and an embedding space
|
|
76
|
+
identified by `modelId` + column width and never by the vendor. What was
|
|
77
|
+
Gemini-bound was the WIRING: `GEMINI_API_KEY` was spelled into three composition
|
|
78
|
+
roots, so a second provider could not obtain a key even though the registry
|
|
79
|
+
would happily build it. Each registry row now names its own key variable, and
|
|
80
|
+
the roots ask, exactly as `instance.md` names the DSN variable rather than
|
|
81
|
+
hardcoding it.
|
|
82
|
+
|
|
83
|
+
`provider: openai` with `model: text-embedding-3-small` and `dim: 1536` reads
|
|
84
|
+
`OPENAI_API_KEY`. Over `fetch`, no SDK — the same call decision 12's 2026-08-22
|
|
85
|
+
revision made for Gemini, and for the same reason.
|
|
86
|
+
|
|
87
|
+
Two things a live call surfaced that a stub would not. Response items carry
|
|
88
|
+
their own `index` and the vendor does not promise array order, so they are
|
|
89
|
+
sorted before the framework pairs them positionally — a shuffled response is the
|
|
90
|
+
same count, the same width and all finite, so every downstream check passes
|
|
91
|
+
while every passage carries another's vector. And an exhausted balance arrives
|
|
92
|
+
as **429**, the same status as a rate limit: it is now read from the vendor's
|
|
93
|
+
`error.type` and never retried, because five exponential backoffs do not add
|
|
94
|
+
credit.
|
|
95
|
+
|
|
96
|
+
Switching provider is a re-embed of the whole corpus and a re-measured
|
|
97
|
+
`vector_floor`. A different provider is a different embedding space, and the
|
|
98
|
+
invariant against copying a calibrated constant applies across vendors with more
|
|
99
|
+
force, not less.
|
|
100
|
+
|
|
101
|
+
Verified live against the real API on a funded key: 1536-dimension vectors, a
|
|
102
|
+
paraphrase at cosine 0.812 against an unrelated sentence at 0.058, and the two
|
|
103
|
+
intents agreeing to 0.9997 — which is the symmetry that makes the empty task
|
|
104
|
+
labels correct. Then through the whole plane: a real record ingested to Postgres
|
|
105
|
+
under `embedding_model = text-embedding-3-small`, 23 chunks, 0 failed, stored at
|
|
106
|
+
the declared width of 1536 and L2-normalized as the framework promises.
|
|
107
|
+
|
|
108
|
+
One more defect the live call found: `buildShippedProvider` handed EVERY
|
|
109
|
+
provider Gemini's task labels from global config, so an OpenAI run logged its
|
|
110
|
+
space as `text-embedding-3-small/d1536/RETRIEVAL_DOCUMENT` — a label that vendor
|
|
111
|
+
has no concept of and never received. The labels moved onto the registry row,
|
|
112
|
+
where a vendor's shape belongs.
|
|
113
|
+
|
|
114
|
+
- eb54871: The npm page shows what the product does, on the first screen.
|
|
115
|
+
|
|
116
|
+
It asserted the headline behaviour — a cited answer, an honest refusal — and
|
|
117
|
+
demonstrated it nowhere. A reader had no way to tell a real mechanism from a
|
|
118
|
+
prompt instruction, which is exactly the skepticism this product exists to
|
|
119
|
+
answer. It now shows three things, all of them real output: the admitted count
|
|
120
|
+
moving when a human approves a draft, the `provenance` and `governance` a search
|
|
121
|
+
hit carries, and an abstention envelope. It also links the hello world.
|
|
122
|
+
|
|
123
|
+
Nothing was added to the tarball; this is the README npm renders.
|
|
124
|
+
|
|
125
|
+
- 6e5ff3e: Repo documentation and a test only — nothing an adopter installs changes.
|
|
126
|
+
|
|
127
|
+
`docs/status.md` named 0.0.42 while the published package was 0.0.53. Authority
|
|
128
|
+
rule 3 makes that file the only authority on what is built, and it is the first
|
|
129
|
+
thing an evaluator's coding agent reads. It is current now, and a docs-truth
|
|
130
|
+
assertion holds it equal to `packages/ksor/package.json` so a Version PR cannot
|
|
131
|
+
bump one without the other. It also records that the full kernel walk was re-run
|
|
132
|
+
against 0.0.53 — it had last run against 0.0.18, thirty-five releases earlier.
|
|
133
|
+
|
|
134
|
+
- ed947c1: The deploy runbook stops ruling out the one step Vercel calls required.
|
|
135
|
+
|
|
136
|
+
`docs/deploying.md` said the silent-404 failure "does not depend on the
|
|
137
|
+
Application Preset". Vercel's own guide says the opposite: a project builds as
|
|
138
|
+
services only when the preset is `Services` AND `vercel.json` carries a
|
|
139
|
+
`services` key, and "if either is missing, Vercel falls back to its default
|
|
140
|
+
framework detection and ignores your services configuration" — which is that
|
|
141
|
+
failure exactly, and no file in the repository can set a project setting.
|
|
142
|
+
|
|
143
|
+
One measurement of ours disagrees with that guide and is recorded rather than
|
|
144
|
+
reconciled: two projects read back from the API, one `Services` and one `Other`,
|
|
145
|
+
both built and served. Both facts are real; guessing between them is what
|
|
146
|
+
produced the sentence that steered adopters away from the fix.
|
|
147
|
+
|
|
148
|
+
The scaffold's runbook now sets the preset at step 2, and ends with the three
|
|
149
|
+
curls that tell a live deployment from a Ready-and-404 one — `/mcp` answering
|
|
150
|
+
405 is the door refusing a GET, which is how you know it is routed at all.
|
|
151
|
+
|
|
3
152
|
## 0.0.53
|
|
4
153
|
|
|
5
154
|
### Patch Changes
|
package/README.md
CHANGED
|
@@ -16,6 +16,36 @@ pnpm install
|
|
|
16
16
|
pnpm dev # the site, live at http://localhost:3000
|
|
17
17
|
```
|
|
18
18
|
|
|
19
|
+
Then write a document and publish it:
|
|
20
|
+
|
|
21
|
+
```console
|
|
22
|
+
$ pnpm exec ksor build
|
|
23
|
+
ksor build: 6 document(s), 5 admitted to a machine surface
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
**Six documents, five admitted.** The one you just wrote is a `draft`, so it
|
|
27
|
+
reaches nothing an AI agent reads — not `llms.txt`, not the markdown twins —
|
|
28
|
+
until a human approves it. Approve it and the count moves. That is the whole
|
|
29
|
+
product, and it costs nothing: no database, no API key, no account.
|
|
30
|
+
|
|
31
|
+
Climb one rung and an agent asks the record a question. The answer carries
|
|
32
|
+
where it came from:
|
|
33
|
+
|
|
34
|
+
```json
|
|
35
|
+
"provenance": { "stable_id": "knowledge/refund-policy", "generation": 1 },
|
|
36
|
+
"governance": { "status": "stable", "approval": { "by": "human:you" } }
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
…and a question the record does not cover is declined rather than guessed at:
|
|
40
|
+
|
|
41
|
+
```json
|
|
42
|
+
{ "ok": false, "abstained": true, "gate": { "floor": 0.622 } }
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
**[Hello world](https://github.com/panaversity/ksor/blob/main/docs/tutorials/00-hello-world.md)**
|
|
46
|
+
walks all of that in about fifteen minutes. Every command and output in it was
|
|
47
|
+
run and pasted as it appeared — including the ones above.
|
|
48
|
+
|
|
19
49
|
One command emits a complete governed project: the record (`knowledge/`,
|
|
20
50
|
plain CommonMark), a working documentation site with hot reload, offline
|
|
21
51
|
search and `llms.txt`, adopter CI, a dependency-free format checker
|
package/dist/cli.mjs
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
import { n as resolveCommand, r as verbs, t as exitCodes } from "./src-dqpI-p1a.mjs";
|
|
3
|
-
import { A as parseViewer, B as
|
|
3
|
+
import { A as parseViewer, B as tallyHandlers, C as contentPoolMin, D as outlineHandler, E as keyRingFromEnv, F as recordIsUndescribed, G as z$1, H as validateViewer, I as runProbe$1, L as searchHandler, M as prewarmPool, N as providerKeyEnv$1, O as parseInstanceText$1, P as readHandler, R as servingPolicy$1, S as contentPool$1, T as instancePathOf$1, U as withPgRetry$1, V as tlsPosture, W as withProbeDeadline$1, _ as assertGovernanceServable$1, a as GovernanceGateError$1, b as checkEmbeddingSpace$1, c as McpServer$1, d as READ_ONLY, f as READ_OUTPUT, g as TextSearchConfigMismatch, h as TRUST_TIERS$1, i as FLOOR, j as pooledEndpointFor, k as parseTrustFloor, l as MissingProviderKeyError$1, m as SchemaVersionError, n as ContentStoreError$1, o as MAX_OUTLINE_LIMIT, p as SEARCH_OUTPUT, r as EmbeddingSpaceMismatch$1, t as AudienceError$1, u as OUTLINE_OUTPUT, v as assertSchemaCompatible, w as embedQueryVlit, x as composeInstructions, y as buildShippedProvider$1, z as storedTextSearchConfig } from "./gateway-api-uhx2l1kC-C2BAxISt.mjs";
|
|
4
4
|
import { appendFileSync, chmodSync, copyFileSync, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, renameSync, rmSync, statSync, unlinkSync, writeFileSync } from "node:fs";
|
|
5
5
|
import { fileURLToPath, pathToFileURL } from "node:url";
|
|
6
6
|
import { InMemoryTransport, LATEST_PROTOCOL_VERSION, createMcpHandler } from "@modelcontextprotocol/server";
|
|
@@ -17,7 +17,7 @@ import { Document, YAMLParseError, isCollection, isMap, isPair, isSeq, parseAllD
|
|
|
17
17
|
import { parseArgs } from "node:util";
|
|
18
18
|
import { readFile } from "node:fs/promises";
|
|
19
19
|
import { execFileSync, spawnSync } from "node:child_process";
|
|
20
|
-
//#region ../content-gateway/dist/main-
|
|
20
|
+
//#region ../content-gateway/dist/main-BHQDvsKA.mjs
|
|
21
21
|
/**
|
|
22
22
|
* The default registration — and the ORIGINAL of the file `ksor init` emits.
|
|
23
23
|
*
|
|
@@ -1008,7 +1008,7 @@ async function compose(rawInstancePath, version) {
|
|
|
1008
1008
|
let provider;
|
|
1009
1009
|
try {
|
|
1010
1010
|
provider = buildShippedProvider$1(instance.embeddingProvider, {
|
|
1011
|
-
apiKey: process.env["
|
|
1011
|
+
apiKey: process.env[providerKeyEnv$1(instance.embeddingProvider) ?? ""] ?? null,
|
|
1012
1012
|
modelId: instance.embeddingModel,
|
|
1013
1013
|
dim: instance.embeddingDim
|
|
1014
1014
|
});
|
|
@@ -4709,7 +4709,7 @@ async function withPgRetry(op, options = {}) {
|
|
|
4709
4709
|
throw lastError;
|
|
4710
4710
|
}
|
|
4711
4711
|
//#endregion
|
|
4712
|
-
//#region ../content/dist/commands-
|
|
4712
|
+
//#region ../content/dist/commands-DM5TajmH.mjs
|
|
4713
4713
|
/**
|
|
4714
4714
|
* EVAL-LOCKED constants, quarried verbatim from the oracle
|
|
4715
4715
|
* (sor-agentfactory @ b554f91, config.py) — changing any of these is a
|
|
@@ -5823,13 +5823,6 @@ const INGEST_RETRY = {
|
|
|
5823
5823
|
initialS: 1,
|
|
5824
5824
|
maxS: 30
|
|
5825
5825
|
};
|
|
5826
|
-
/** Fail-fast for the query path: 2 attempts, sub-second backoff, no 429 retry — worst case a
|
|
5827
|
-
* query embed costs ~1 s before the caller degrades, never a 15 s stall on the read. */
|
|
5828
|
-
const QUERY_RETRY = {
|
|
5829
|
-
attempts: 2,
|
|
5830
|
-
initialS: .2,
|
|
5831
|
-
maxS: .5
|
|
5832
|
-
};
|
|
5833
5826
|
function sleepS(seconds) {
|
|
5834
5827
|
return new Promise((resolve) => setTimeout(resolve, seconds * 1e3));
|
|
5835
5828
|
}
|
|
@@ -5855,17 +5848,6 @@ async function embedIntent(texts, opts) {
|
|
|
5855
5848
|
return contract(texts, await withRetry(INGEST_RETRY, (exc) => provider.isRetryable(exc), () => provider.reset(), () => provider.embed(texts, { intent })), provider.dim);
|
|
5856
5849
|
}
|
|
5857
5850
|
/**
|
|
5858
|
-
* The READ plane's door (query-embed, intent "query"). Same contract;
|
|
5859
|
-
* FAIL-FAST retry (no 429, 2 attempts): a rate limit degrades the search to
|
|
5860
|
-
* keyword-only in under a second instead of stalling the read ~15 s. Async
|
|
5861
|
-
* like everything here — the oracle's `a` prefix marked Python's async
|
|
5862
|
-
* flavor and is kept only for greppability against the Python.
|
|
5863
|
-
*/
|
|
5864
|
-
async function aembedIntent(texts, opts) {
|
|
5865
|
-
const { provider, intent } = opts;
|
|
5866
|
-
return contract(texts, await withRetry(QUERY_RETRY, (exc) => provider.isRetryableQuery(exc), () => provider.reset(), () => provider.embed(texts, { intent })), provider.dim);
|
|
5867
|
-
}
|
|
5868
|
-
/**
|
|
5869
5851
|
* The deterministic, key-free fake provider — a KSOR ADDITION (no oracle
|
|
5870
5852
|
* counterpart; decision 6 conversion note). It exists so the DB tier and CI
|
|
5871
5853
|
* can exercise ingest + retrieval end to end without a vendor key: same text
|
|
@@ -5937,7 +5919,7 @@ var FakeEmbeddingProvider = class {
|
|
|
5937
5919
|
}
|
|
5938
5920
|
reset() {}
|
|
5939
5921
|
};
|
|
5940
|
-
const DEFAULT_BASE = "https://generativelanguage.googleapis.com/v1beta";
|
|
5922
|
+
const DEFAULT_BASE$1 = "https://generativelanguage.googleapis.com/v1beta";
|
|
5941
5923
|
/**
|
|
5942
5924
|
* An HTTP-shaped failure carrying the status the retry classifier reads.
|
|
5943
5925
|
*
|
|
@@ -5955,7 +5937,7 @@ var GeminiHttpError = class extends Error {
|
|
|
5955
5937
|
};
|
|
5956
5938
|
/** One POST, with the key in a HEADER — never the query string, which is logged. */
|
|
5957
5939
|
async function post(opts, apiKey, path, body, timeoutMs) {
|
|
5958
|
-
const res = await (opts.fetchImpl ?? fetch)(`${opts.baseUrl ?? DEFAULT_BASE}${path}`, {
|
|
5940
|
+
const res = await (opts.fetchImpl ?? fetch)(`${opts.baseUrl ?? DEFAULT_BASE$1}${path}`, {
|
|
5959
5941
|
method: "POST",
|
|
5960
5942
|
headers: {
|
|
5961
5943
|
"x-goog-api-key": apiKey,
|
|
@@ -6025,33 +6007,33 @@ function geminiRestTextClient(apiKey, opts = {}) {
|
|
|
6025
6007
|
* Python-SDK failure mode with no JS equivalent; `reset()`
|
|
6026
6008
|
* keeps its drop-never-close contract regardless.
|
|
6027
6009
|
*/
|
|
6028
|
-
function httpStatusOf(exc) {
|
|
6010
|
+
function httpStatusOf$1(exc) {
|
|
6029
6011
|
if (!(exc instanceof Error)) return void 0;
|
|
6030
6012
|
const status = exc.status;
|
|
6031
6013
|
return typeof status === "number" ? status : void 0;
|
|
6032
6014
|
}
|
|
6033
|
-
function isTransportBlip(exc, depth = 0) {
|
|
6015
|
+
function isTransportBlip$1(exc, depth = 0) {
|
|
6034
6016
|
if (depth > 5 || !(exc instanceof Error)) return false;
|
|
6035
6017
|
if (exc.name === "AbortError" || exc.name === "TimeoutError") return true;
|
|
6036
6018
|
const code = exc.code;
|
|
6037
6019
|
if (typeof code === "string" && /^E[A-Z0-9_]+$/.test(code)) return true;
|
|
6038
6020
|
if (exc instanceof TypeError && exc.message.includes("fetch failed")) return true;
|
|
6039
|
-
return isTransportBlip(exc.cause, depth + 1);
|
|
6021
|
+
return isTransportBlip$1(exc.cause, depth + 1);
|
|
6040
6022
|
}
|
|
6041
6023
|
/** The Gemini taxonomy for the INGEST plane: transport blips, 5xx, AND 429 —
|
|
6042
6024
|
* batch work is resumable and has no user waiting. */
|
|
6043
|
-
function isRetryable(exc) {
|
|
6044
|
-
if (isTransportBlip(exc)) return true;
|
|
6045
|
-
const status = httpStatusOf(exc);
|
|
6025
|
+
function isRetryable$1(exc) {
|
|
6026
|
+
if (isTransportBlip$1(exc)) return true;
|
|
6027
|
+
const status = httpStatusOf$1(exc);
|
|
6046
6028
|
if (status === void 0) return false;
|
|
6047
6029
|
return status >= 500 && status <= 599 || status === 429;
|
|
6048
6030
|
}
|
|
6049
6031
|
/** The READ path's predicate: transport blips + 5xx only — NEVER 429. A
|
|
6050
6032
|
* rate-limited project stays rate-limited on the next second; the correct
|
|
6051
6033
|
* move is to degrade to keyword-only NOW, not stall the read behind backoff. */
|
|
6052
|
-
function isRetryableQuery(exc) {
|
|
6053
|
-
if (isTransportBlip(exc)) return true;
|
|
6054
|
-
const status = httpStatusOf(exc);
|
|
6034
|
+
function isRetryableQuery$1(exc) {
|
|
6035
|
+
if (isTransportBlip$1(exc)) return true;
|
|
6036
|
+
const status = httpStatusOf$1(exc);
|
|
6055
6037
|
return status !== void 0 && status >= 500 && status <= 599;
|
|
6056
6038
|
}
|
|
6057
6039
|
/**
|
|
@@ -6102,10 +6084,10 @@ var GeminiEmbeddingProvider = class {
|
|
|
6102
6084
|
})).embeddings ?? []).map((e) => [...e.values ?? []]);
|
|
6103
6085
|
}
|
|
6104
6086
|
isRetryable(exc) {
|
|
6105
|
-
return isRetryable(exc);
|
|
6087
|
+
return isRetryable$1(exc);
|
|
6106
6088
|
}
|
|
6107
6089
|
isRetryableQuery(exc) {
|
|
6108
|
-
return isRetryableQuery(exc);
|
|
6090
|
+
return isRetryableQuery$1(exc);
|
|
6109
6091
|
}
|
|
6110
6092
|
};
|
|
6111
6093
|
/**
|
|
@@ -6118,7 +6100,7 @@ var GeminiTextGenerator = class {
|
|
|
6118
6100
|
clientFactory;
|
|
6119
6101
|
client = null;
|
|
6120
6102
|
constructor(opts) {
|
|
6121
|
-
this.model = opts.model ?? "gemini-
|
|
6103
|
+
this.model = opts.model ?? "gemini-3.7-flash";
|
|
6122
6104
|
this.clientFactory = opts.clientFactory ?? (() => geminiRestTextClient(opts.apiKey));
|
|
6123
6105
|
}
|
|
6124
6106
|
getClient() {
|
|
@@ -6139,9 +6121,163 @@ var GeminiTextGenerator = class {
|
|
|
6139
6121
|
}
|
|
6140
6122
|
})).text ?? "";
|
|
6141
6123
|
}
|
|
6124
|
+
isRetryable(exc) {
|
|
6125
|
+
return isRetryable$1(exc);
|
|
6126
|
+
}
|
|
6127
|
+
};
|
|
6128
|
+
/**
|
|
6129
|
+
* OpenAI's embedding endpoint, over `fetch` — no SDK.
|
|
6130
|
+
*
|
|
6131
|
+
* The same call decision 12's 2026-08-22 revision made for Gemini, for the same
|
|
6132
|
+
* reason: this is ONE HTTP call behind a structurally-typed slice, and a vendor
|
|
6133
|
+
* SDK would put megabytes and a transitive dependency tree into every
|
|
6134
|
+
* `ksor init` that never embeds anything. If a provider ever needs an SDK, the
|
|
6135
|
+
* seam takes one through `clientFactory`; nothing here forecloses that.
|
|
6136
|
+
*
|
|
6137
|
+
* WHAT DIFFERS FROM GEMINI, and it is worth knowing before choosing:
|
|
6138
|
+
*
|
|
6139
|
+
* Gemini embeds ASYMMETRICALLY — `RETRIEVAL_DOCUMENT` and `RETRIEVAL_QUERY`
|
|
6140
|
+
* produce different vectors for the same text, and the plane's intent picks
|
|
6141
|
+
* which. OpenAI has no task type: a query and a document are embedded
|
|
6142
|
+
* identically. The seam anticipates exactly this — "a provider whose two
|
|
6143
|
+
* vendor labels are equal can never mis-route a plane"
|
|
6144
|
+
* (`lib/embedding.ts`) — so both labels are the empty string and the intent
|
|
6145
|
+
* reaches this transport and is deliberately ignored.
|
|
6146
|
+
*
|
|
6147
|
+
* `dimensions` is supported on `text-embedding-3-*` only. Asking an older
|
|
6148
|
+
* model for a dimension is an error there, not a silent full-width vector,
|
|
6149
|
+
* which is the failure the framework's own width check would catch anyway.
|
|
6150
|
+
*/
|
|
6151
|
+
/**
|
|
6152
|
+
* An HTTP-shaped failure carrying the status the retry classifier reads — and
|
|
6153
|
+
* the vendor's own error `type`, which the status alone does not distinguish.
|
|
6154
|
+
*
|
|
6155
|
+
* OpenAI answers a spent balance with **429**, the same status as a rate limit:
|
|
6156
|
+
* `{"error":{"type":"insufficient_quota","code":"credit_balance_exhausted"}}`
|
|
6157
|
+
* (observed live, 2026-09-01). One clears by waiting and one never will, so a
|
|
6158
|
+
* classifier reading only the status retries a billing problem five times with
|
|
6159
|
+
* exponential backoff and then reports it — slowly, and as if it had been
|
|
6160
|
+
* transient.
|
|
6161
|
+
*/
|
|
6162
|
+
var OpenAiHttpError = class extends Error {
|
|
6163
|
+
status;
|
|
6164
|
+
/** The vendor's `error.type`, when it sends one. */
|
|
6165
|
+
kind;
|
|
6166
|
+
constructor(status, detail, kind = null) {
|
|
6167
|
+
super(`OpenAI API error ${status}: ${detail}`);
|
|
6168
|
+
this.name = "OpenAiHttpError";
|
|
6169
|
+
this.status = status;
|
|
6170
|
+
this.kind = kind;
|
|
6171
|
+
}
|
|
6172
|
+
};
|
|
6173
|
+
const DEFAULT_BASE = "https://api.openai.com/v1";
|
|
6174
|
+
function openAiRestEmbedClient(apiKey, opts = {}) {
|
|
6175
|
+
return { async embed(params) {
|
|
6176
|
+
const res = await (opts.fetchImpl ?? fetch)(`${opts.baseUrl ?? DEFAULT_BASE}/embeddings`, {
|
|
6177
|
+
method: "POST",
|
|
6178
|
+
headers: {
|
|
6179
|
+
authorization: `Bearer ${apiKey}`,
|
|
6180
|
+
"content-type": "application/json"
|
|
6181
|
+
},
|
|
6182
|
+
body: JSON.stringify({
|
|
6183
|
+
model: params.model,
|
|
6184
|
+
input: [...params.input],
|
|
6185
|
+
dimensions: params.dimensions
|
|
6186
|
+
}),
|
|
6187
|
+
signal: AbortSignal.timeout(params.timeoutMs)
|
|
6188
|
+
});
|
|
6189
|
+
const text = await res.text();
|
|
6190
|
+
if (!res.ok) {
|
|
6191
|
+
let detail = text.slice(0, 300);
|
|
6192
|
+
let kind = null;
|
|
6193
|
+
try {
|
|
6194
|
+
const err = JSON.parse(text).error;
|
|
6195
|
+
if (typeof err?.message === "string") detail = err.message;
|
|
6196
|
+
if (typeof err?.type === "string") kind = err.type;
|
|
6197
|
+
} catch {}
|
|
6198
|
+
throw new OpenAiHttpError(res.status, detail, kind);
|
|
6199
|
+
}
|
|
6200
|
+
const data = [...JSON.parse(text).data ?? []];
|
|
6201
|
+
data.sort((a, b) => (a.index ?? 0) - (b.index ?? 0));
|
|
6202
|
+
return { embeddings: data.map((d) => ({ values: d.embedding })) };
|
|
6203
|
+
} };
|
|
6204
|
+
}
|
|
6205
|
+
/** True for a transport blip with no HTTP status of its own. */
|
|
6206
|
+
function isTransportBlip(exc) {
|
|
6207
|
+
if (exc instanceof OpenAiHttpError) return false;
|
|
6208
|
+
const name = exc?.name;
|
|
6209
|
+
return name === "AbortError" || name === "TimeoutError" || name === "TypeError";
|
|
6210
|
+
}
|
|
6211
|
+
function httpStatusOf(exc) {
|
|
6212
|
+
return exc instanceof OpenAiHttpError ? exc.status : void 0;
|
|
6213
|
+
}
|
|
6214
|
+
/**
|
|
6215
|
+
* The INGEST plane's taxonomy: transport blips, 5xx, AND 429 — batch work is
|
|
6216
|
+
* resumable and has nobody waiting. Deliberately the same shape as the Gemini
|
|
6217
|
+
* adapter's, because the two planes are a property of ksor, not of a vendor.
|
|
6218
|
+
*/
|
|
6219
|
+
function isRetryable(exc) {
|
|
6220
|
+
if (isTransportBlip(exc)) return true;
|
|
6221
|
+
if (exc instanceof OpenAiHttpError && exc.kind === "insufficient_quota") return false;
|
|
6222
|
+
const status = httpStatusOf(exc);
|
|
6223
|
+
if (status === void 0) return false;
|
|
6224
|
+
return status >= 500 && status <= 599 || status === 429;
|
|
6225
|
+
}
|
|
6226
|
+
/**
|
|
6227
|
+
* The READ plane's: transport blips + 5xx only, NEVER 429. A rate-limited
|
|
6228
|
+
* project stays rate-limited on the next second, so a search degrades to
|
|
6229
|
+
* keyword-only now rather than stalling a reader behind backoff.
|
|
6230
|
+
*/
|
|
6231
|
+
function isRetryableQuery(exc) {
|
|
6232
|
+
if (isTransportBlip(exc)) return true;
|
|
6233
|
+
const status = httpStatusOf(exc);
|
|
6234
|
+
return status !== void 0 && status >= 500 && status <= 599;
|
|
6235
|
+
}
|
|
6236
|
+
var OpenAiEmbeddingProvider = class {
|
|
6237
|
+
providerId = "openai";
|
|
6238
|
+
modelId;
|
|
6239
|
+
dim;
|
|
6240
|
+
documentTaskLabel;
|
|
6241
|
+
queryTaskLabel;
|
|
6242
|
+
documentTimeoutMs;
|
|
6243
|
+
queryTimeoutMs;
|
|
6244
|
+
clientFactory;
|
|
6245
|
+
client = null;
|
|
6246
|
+
constructor(opts) {
|
|
6247
|
+
this.modelId = opts.modelId;
|
|
6248
|
+
this.dim = opts.dim;
|
|
6249
|
+
this.documentTaskLabel = opts.documentTaskLabel;
|
|
6250
|
+
this.queryTaskLabel = opts.queryTaskLabel;
|
|
6251
|
+
this.documentTimeoutMs = Math.trunc(opts.documentTimeoutS * 1e3);
|
|
6252
|
+
this.queryTimeoutMs = Math.trunc(opts.queryTimeoutS * 1e3);
|
|
6253
|
+
this.clientFactory = opts.clientFactory ?? (() => openAiRestEmbedClient(opts.apiKey));
|
|
6254
|
+
}
|
|
6255
|
+
get recipe() {
|
|
6256
|
+
return `${this.modelId}/d${this.dim}/${this.documentTaskLabel}`;
|
|
6257
|
+
}
|
|
6258
|
+
getClient() {
|
|
6259
|
+
this.client ??= this.clientFactory();
|
|
6260
|
+
return this.client;
|
|
6261
|
+
}
|
|
6262
|
+
/** DROP the client reference, never close it: in-flight calls finish on their
|
|
6263
|
+
* own reference and the next call rebuilds lazily. Idempotent. */
|
|
6264
|
+
reset() {
|
|
6265
|
+
this.client = null;
|
|
6266
|
+
}
|
|
6267
|
+
async embed(texts, opts) {
|
|
6268
|
+
return (await this.getClient().embed({
|
|
6269
|
+
model: this.modelId,
|
|
6270
|
+
input: texts,
|
|
6271
|
+
dimensions: this.dim,
|
|
6272
|
+
timeoutMs: opts.intent === "document" ? this.documentTimeoutMs : this.queryTimeoutMs
|
|
6273
|
+
})).embeddings.map((e) => [...e.values ?? []]);
|
|
6274
|
+
}
|
|
6142
6275
|
isRetryable(exc) {
|
|
6143
6276
|
return isRetryable(exc);
|
|
6144
6277
|
}
|
|
6278
|
+
isRetryableQuery(exc) {
|
|
6279
|
+
return isRetryableQuery(exc);
|
|
6280
|
+
}
|
|
6145
6281
|
};
|
|
6146
6282
|
/**
|
|
6147
6283
|
* The embedding-provider registry — a plain object, NOT any discovery
|
|
@@ -6177,11 +6313,30 @@ var MissingProviderKeyError = class extends Error {
|
|
|
6177
6313
|
const PROVIDERS = {
|
|
6178
6314
|
gemini: {
|
|
6179
6315
|
build: (opts) => new GeminiEmbeddingProvider(opts),
|
|
6180
|
-
needsApiKey: true
|
|
6316
|
+
needsApiKey: true,
|
|
6317
|
+
keyEnv: "GEMINI_API_KEY",
|
|
6318
|
+
taskLabels: {
|
|
6319
|
+
document: EMBED_TASK_DOCUMENT,
|
|
6320
|
+
query: EMBED_TASK_QUERY
|
|
6321
|
+
}
|
|
6322
|
+
},
|
|
6323
|
+
openai: {
|
|
6324
|
+
build: (opts) => new OpenAiEmbeddingProvider(opts),
|
|
6325
|
+
needsApiKey: true,
|
|
6326
|
+
keyEnv: "OPENAI_API_KEY",
|
|
6327
|
+
taskLabels: {
|
|
6328
|
+
document: "",
|
|
6329
|
+
query: ""
|
|
6330
|
+
}
|
|
6181
6331
|
},
|
|
6182
6332
|
fake: {
|
|
6183
6333
|
build: (opts) => new FakeEmbeddingProvider(opts),
|
|
6184
|
-
needsApiKey: false
|
|
6334
|
+
needsApiKey: false,
|
|
6335
|
+
keyEnv: null,
|
|
6336
|
+
taskLabels: {
|
|
6337
|
+
document: EMBED_TASK_DOCUMENT,
|
|
6338
|
+
query: EMBED_TASK_QUERY
|
|
6339
|
+
}
|
|
6185
6340
|
}
|
|
6186
6341
|
};
|
|
6187
6342
|
function entryFor(name) {
|
|
@@ -6196,6 +6351,13 @@ function providerNeedsApiKey(name) {
|
|
|
6196
6351
|
return entryFor(name).needsApiKey;
|
|
6197
6352
|
}
|
|
6198
6353
|
/**
|
|
6354
|
+
* The environment variable this provider's key comes from, or null when it
|
|
6355
|
+
* needs none. Unknown name → the same loud error as building it.
|
|
6356
|
+
*/
|
|
6357
|
+
function providerKeyEnv(name) {
|
|
6358
|
+
return entryFor(name).keyEnv;
|
|
6359
|
+
}
|
|
6360
|
+
/**
|
|
6199
6361
|
* The port door: the named provider bound to the DECLARED embedding space and
|
|
6200
6362
|
* the framework's timeout knobs. `modelId`/`dim` omitted = the shipped
|
|
6201
6363
|
* config space (EMBED_MODEL / EMBED_DIM, eval-locked); an instance may
|
|
@@ -6212,8 +6374,8 @@ function buildShippedProvider(name, opts) {
|
|
|
6212
6374
|
apiKey: opts.apiKey ?? "",
|
|
6213
6375
|
modelId: opts.modelId ?? "gemini-embedding-001",
|
|
6214
6376
|
dim: opts.dim ?? 1536,
|
|
6215
|
-
documentTaskLabel:
|
|
6216
|
-
queryTaskLabel:
|
|
6377
|
+
documentTaskLabel: entry.taskLabels.document,
|
|
6378
|
+
queryTaskLabel: entry.taskLabels.query,
|
|
6217
6379
|
documentTimeoutS: EMBED_TIMEOUT_S(),
|
|
6218
6380
|
queryTimeoutS: QUERY_EMBED_TIMEOUT_S()
|
|
6219
6381
|
});
|
|
@@ -6711,7 +6873,7 @@ function calibrationScope(viewer) {
|
|
|
6711
6873
|
async function scoreQueries(pool, scope, gucs, provider, queries, inCorpus) {
|
|
6712
6874
|
const out = [];
|
|
6713
6875
|
for (const query of queries) {
|
|
6714
|
-
const [vector] = await
|
|
6876
|
+
const [vector] = await embedIntent([query], {
|
|
6715
6877
|
provider,
|
|
6716
6878
|
intent: "query"
|
|
6717
6879
|
});
|
|
@@ -9743,6 +9905,43 @@ function pct(fraction) {
|
|
|
9743
9905
|
return `${(fraction * 100).toFixed(0)}%`;
|
|
9744
9906
|
}
|
|
9745
9907
|
/**
|
|
9908
|
+
* What to do when calibration is refused for quota — named, not left as the
|
|
9909
|
+
* vendor's sentence.
|
|
9910
|
+
*
|
|
9911
|
+
* Both of these were hit walking a real free-tier key, and they are DIFFERENT
|
|
9912
|
+
* failures with different remedies, which is why the raw error is not enough:
|
|
9913
|
+
*
|
|
9914
|
+
* generate_content …/gemini-3.7-flash, limit 5 the SYNTHESIZED door writes
|
|
9915
|
+
* one probe question per
|
|
9916
|
+
* sampled passage, and a free
|
|
9917
|
+
* key allows a handful of
|
|
9918
|
+
* generations a minute. No
|
|
9919
|
+
* amount of waiting fixes a
|
|
9920
|
+
* corpus of any size; the
|
|
9921
|
+
* answer is the zero-LLM door.
|
|
9922
|
+
* global_embed_content_requests_per_minute the EMBEDDING endpoint, which
|
|
9923
|
+
* both doors use. Transient:
|
|
9924
|
+
* an ingest immediately before
|
|
9925
|
+
* a calibration spends the same
|
|
9926
|
+
* per-minute budget.
|
|
9927
|
+
*
|
|
9928
|
+
* Product principle 4: a failure states what is wrong, why the rule exists, and
|
|
9929
|
+
* how to fix it. The vendor's message states only the first.
|
|
9930
|
+
*/
|
|
9931
|
+
/** The generation quota, which no wait resolves on a free key. */
|
|
9932
|
+
const GENERATION = /generate_content|generativelanguage\.googleapis\.com\/generate/i;
|
|
9933
|
+
/** The embedding quota, which clears on its own. */
|
|
9934
|
+
const EMBEDDING = /embed_content|global_embed/i;
|
|
9935
|
+
/**
|
|
9936
|
+
* The remedy for a quota refusal, or null when the failure is not one this
|
|
9937
|
+
* knows — in which case the vendor's own message stands, unembellished.
|
|
9938
|
+
*/
|
|
9939
|
+
function quotaRemedy(message) {
|
|
9940
|
+
if (GENERATION.test(message)) return "the SYNTHESIZED door writes one probe question per sampled passage with an LLM, and a free-tier key allows only a few generations a minute — a bigger corpus makes this worse, not better.\n fix: calibrate with zero LLM — write your in-corpus questions one per line and pass\n --queries-file PATH. The floor is measured the same way; only the questions differ,\n and the door is recorded beside the number so the two are never compared.";
|
|
9941
|
+
if (EMBEDDING.test(message)) return "the EMBEDDING endpoint is limited per minute, and both doors use it — an ingest immediately before this spends the same budget.\n fix: wait about a minute and run it again. Nothing was written; calibration only reads.";
|
|
9942
|
+
return null;
|
|
9943
|
+
}
|
|
9944
|
+
/**
|
|
9746
9945
|
* How close to the floor an answer has to be to count as marginal.
|
|
9747
9946
|
*
|
|
9748
9947
|
* 0.01 is not a tuned threshold — it is the size of the decision in this
|
|
@@ -10175,8 +10374,9 @@ function composeProvider(instance) {
|
|
|
10175
10374
|
try {
|
|
10176
10375
|
let apiKey = null;
|
|
10177
10376
|
if (providerNeedsApiKey(instance.embeddingProvider)) {
|
|
10178
|
-
|
|
10179
|
-
|
|
10377
|
+
const keyEnv = providerKeyEnv(instance.embeddingProvider) ?? "";
|
|
10378
|
+
apiKey = process.env[keyEnv] || null;
|
|
10379
|
+
if (apiKey === null) return fail$1(ENVIRONMENT, `${keyEnv} is required (the instance's embedding provider ${JSON.stringify(instance.embeddingProvider)} needs a key)\n fix: export ${keyEnv}=... and rerun`);
|
|
10180
10380
|
}
|
|
10181
10381
|
return buildShippedProvider(instance.embeddingProvider, {
|
|
10182
10382
|
apiKey,
|
|
@@ -10437,7 +10637,7 @@ async function calibrateCommand(args) {
|
|
|
10437
10637
|
textGenerator = new GeminiTextGenerator({ apiKey });
|
|
10438
10638
|
}
|
|
10439
10639
|
const ooc = values["ooc-file"] === void 0 ? null : parseQueriesFile(readFileSync(values["ooc-file"], "utf8"));
|
|
10440
|
-
const report = await withPool(dsn, async (pool) => runCalibration(pool, {
|
|
10640
|
+
const report = await withQuotaRemedy(async () => withPool(dsn, async (pool) => runCalibration(pool, {
|
|
10441
10641
|
tenantId: instance.tenantId,
|
|
10442
10642
|
corpusId: instance.corpusId,
|
|
10443
10643
|
viewer: await widestViewer(pool, instance),
|
|
@@ -10448,12 +10648,31 @@ async function calibrateCommand(args) {
|
|
|
10448
10648
|
oocProbes: ooc,
|
|
10449
10649
|
perNode: values["per-node"] === void 0 ? void 0 : intFlag("--per-node", values["per-node"]),
|
|
10450
10650
|
minChars: values["min-chars"] === void 0 ? void 0 : intFlag("--min-chars", values["min-chars"])
|
|
10451
|
-
}));
|
|
10651
|
+
})));
|
|
10452
10652
|
process.stdout.write(renderReport(report, GATE_PREDICATE_DIGEST) + "\n");
|
|
10453
10653
|
const advice = overlapAdvice(report);
|
|
10454
10654
|
if (advice !== null) process.stdout.write(advice);
|
|
10455
10655
|
return 0;
|
|
10456
10656
|
}
|
|
10657
|
+
/**
|
|
10658
|
+
* Run a calibration, turning a quota refusal into the remedy for THAT quota.
|
|
10659
|
+
*
|
|
10660
|
+
* Both failures reach here as the vendor's own sentence, which states what is
|
|
10661
|
+
* wrong and neither why nor how to fix it — and the two need opposite answers
|
|
10662
|
+
* (change door vs wait a minute). Anything `quotaRemedy` does not recognise is
|
|
10663
|
+
* re-thrown untouched: inventing advice for an error nobody has read is worse
|
|
10664
|
+
* than passing the vendor's through.
|
|
10665
|
+
*/
|
|
10666
|
+
async function withQuotaRemedy(op) {
|
|
10667
|
+
try {
|
|
10668
|
+
return await op();
|
|
10669
|
+
} catch (exc) {
|
|
10670
|
+
const message = exc instanceof Error ? exc.message : String(exc);
|
|
10671
|
+
const remedy = quotaRemedy(message);
|
|
10672
|
+
if (remedy === null) throw exc;
|
|
10673
|
+
throw Object.assign(/* @__PURE__ */ new Error(`${message}\n why: ${remedy}`), { slug: "ksor-calibrate-quota" });
|
|
10674
|
+
}
|
|
10675
|
+
}
|
|
10457
10676
|
/** How many days of traffic one --check reads. Bounded so a busy record cannot make it expensive. */
|
|
10458
10677
|
const DRIFT_DEFAULT_DAYS = 30;
|
|
10459
10678
|
/**
|
|
@@ -13350,7 +13569,8 @@ function isEnvironmentError(value) {
|
|
|
13350
13569
|
const EMITTED_NAMES = /* @__PURE__ */ new Map([
|
|
13351
13570
|
["gitignore", ".gitignore"],
|
|
13352
13571
|
["env.example", ".env.example"],
|
|
13353
|
-
["dockerignore", ".dockerignore"]
|
|
13572
|
+
["dockerignore", ".dockerignore"],
|
|
13573
|
+
["mcp.json", ".mcp.json"]
|
|
13354
13574
|
]);
|
|
13355
13575
|
const TEXT_EXTENSIONS = /* @__PURE__ */ new Set([
|
|
13356
13576
|
".md",
|
|
@@ -6,7 +6,7 @@ import { z, z as z$1 } from "zod";
|
|
|
6
6
|
import path, { join } from "node:path";
|
|
7
7
|
import { createHash, createHmac, randomBytes, timingSafeEqual } from "node:crypto";
|
|
8
8
|
import pg from "pg";
|
|
9
|
-
//#region ../content-gateway/dist/gateway-api-
|
|
9
|
+
//#region ../content-gateway/dist/gateway-api-uhx2l1kC.mjs
|
|
10
10
|
var __commonJSMin = (cb, mod) => () => (mod || (cb((mod = { exports: {} }).exports, mod), cb = null), mod.exports);
|
|
11
11
|
var __require = /* #__PURE__ */ (() => createRequire(import.meta.url))();
|
|
12
12
|
/**
|
|
@@ -8899,7 +8899,7 @@ var FakeEmbeddingProvider = class {
|
|
|
8899
8899
|
}
|
|
8900
8900
|
reset() {}
|
|
8901
8901
|
};
|
|
8902
|
-
const DEFAULT_BASE = "https://generativelanguage.googleapis.com/v1beta";
|
|
8902
|
+
const DEFAULT_BASE$1 = "https://generativelanguage.googleapis.com/v1beta";
|
|
8903
8903
|
/**
|
|
8904
8904
|
* An HTTP-shaped failure carrying the status the retry classifier reads.
|
|
8905
8905
|
*
|
|
@@ -8917,7 +8917,7 @@ var GeminiHttpError = class extends Error {
|
|
|
8917
8917
|
};
|
|
8918
8918
|
/** One POST, with the key in a HEADER — never the query string, which is logged. */
|
|
8919
8919
|
async function post(opts, apiKey, path, body, timeoutMs) {
|
|
8920
|
-
const res = await (opts.fetchImpl ?? fetch)(`${opts.baseUrl ?? DEFAULT_BASE}${path}`, {
|
|
8920
|
+
const res = await (opts.fetchImpl ?? fetch)(`${opts.baseUrl ?? DEFAULT_BASE$1}${path}`, {
|
|
8921
8921
|
method: "POST",
|
|
8922
8922
|
headers: {
|
|
8923
8923
|
"x-goog-api-key": apiKey,
|
|
@@ -8973,33 +8973,33 @@ function geminiRestEmbedClient(apiKey, opts = {}) {
|
|
|
8973
8973
|
* Python-SDK failure mode with no JS equivalent; `reset()`
|
|
8974
8974
|
* keeps its drop-never-close contract regardless.
|
|
8975
8975
|
*/
|
|
8976
|
-
function httpStatusOf(exc) {
|
|
8976
|
+
function httpStatusOf$1(exc) {
|
|
8977
8977
|
if (!(exc instanceof Error)) return void 0;
|
|
8978
8978
|
const status = exc.status;
|
|
8979
8979
|
return typeof status === "number" ? status : void 0;
|
|
8980
8980
|
}
|
|
8981
|
-
function isTransportBlip(exc, depth = 0) {
|
|
8981
|
+
function isTransportBlip$1(exc, depth = 0) {
|
|
8982
8982
|
if (depth > 5 || !(exc instanceof Error)) return false;
|
|
8983
8983
|
if (exc.name === "AbortError" || exc.name === "TimeoutError") return true;
|
|
8984
8984
|
const code = exc.code;
|
|
8985
8985
|
if (typeof code === "string" && /^E[A-Z0-9_]+$/.test(code)) return true;
|
|
8986
8986
|
if (exc instanceof TypeError && exc.message.includes("fetch failed")) return true;
|
|
8987
|
-
return isTransportBlip(exc.cause, depth + 1);
|
|
8987
|
+
return isTransportBlip$1(exc.cause, depth + 1);
|
|
8988
8988
|
}
|
|
8989
8989
|
/** The Gemini taxonomy for the INGEST plane: transport blips, 5xx, AND 429 —
|
|
8990
8990
|
* batch work is resumable and has no user waiting. */
|
|
8991
|
-
function isRetryable(exc) {
|
|
8992
|
-
if (isTransportBlip(exc)) return true;
|
|
8993
|
-
const status = httpStatusOf(exc);
|
|
8991
|
+
function isRetryable$1(exc) {
|
|
8992
|
+
if (isTransportBlip$1(exc)) return true;
|
|
8993
|
+
const status = httpStatusOf$1(exc);
|
|
8994
8994
|
if (status === void 0) return false;
|
|
8995
8995
|
return status >= 500 && status <= 599 || status === 429;
|
|
8996
8996
|
}
|
|
8997
8997
|
/** The READ path's predicate: transport blips + 5xx only — NEVER 429. A
|
|
8998
8998
|
* rate-limited project stays rate-limited on the next second; the correct
|
|
8999
8999
|
* move is to degrade to keyword-only NOW, not stall the read behind backoff. */
|
|
9000
|
-
function isRetryableQuery(exc) {
|
|
9001
|
-
if (isTransportBlip(exc)) return true;
|
|
9002
|
-
const status = httpStatusOf(exc);
|
|
9000
|
+
function isRetryableQuery$1(exc) {
|
|
9001
|
+
if (isTransportBlip$1(exc)) return true;
|
|
9002
|
+
const status = httpStatusOf$1(exc);
|
|
9003
9003
|
return status !== void 0 && status >= 500 && status <= 599;
|
|
9004
9004
|
}
|
|
9005
9005
|
/**
|
|
@@ -9049,6 +9049,160 @@ var GeminiEmbeddingProvider = class {
|
|
|
9049
9049
|
}
|
|
9050
9050
|
})).embeddings ?? []).map((e) => [...e.values ?? []]);
|
|
9051
9051
|
}
|
|
9052
|
+
isRetryable(exc) {
|
|
9053
|
+
return isRetryable$1(exc);
|
|
9054
|
+
}
|
|
9055
|
+
isRetryableQuery(exc) {
|
|
9056
|
+
return isRetryableQuery$1(exc);
|
|
9057
|
+
}
|
|
9058
|
+
};
|
|
9059
|
+
/**
|
|
9060
|
+
* OpenAI's embedding endpoint, over `fetch` — no SDK.
|
|
9061
|
+
*
|
|
9062
|
+
* The same call decision 12's 2026-08-22 revision made for Gemini, for the same
|
|
9063
|
+
* reason: this is ONE HTTP call behind a structurally-typed slice, and a vendor
|
|
9064
|
+
* SDK would put megabytes and a transitive dependency tree into every
|
|
9065
|
+
* `ksor init` that never embeds anything. If a provider ever needs an SDK, the
|
|
9066
|
+
* seam takes one through `clientFactory`; nothing here forecloses that.
|
|
9067
|
+
*
|
|
9068
|
+
* WHAT DIFFERS FROM GEMINI, and it is worth knowing before choosing:
|
|
9069
|
+
*
|
|
9070
|
+
* Gemini embeds ASYMMETRICALLY — `RETRIEVAL_DOCUMENT` and `RETRIEVAL_QUERY`
|
|
9071
|
+
* produce different vectors for the same text, and the plane's intent picks
|
|
9072
|
+
* which. OpenAI has no task type: a query and a document are embedded
|
|
9073
|
+
* identically. The seam anticipates exactly this — "a provider whose two
|
|
9074
|
+
* vendor labels are equal can never mis-route a plane"
|
|
9075
|
+
* (`lib/embedding.ts`) — so both labels are the empty string and the intent
|
|
9076
|
+
* reaches this transport and is deliberately ignored.
|
|
9077
|
+
*
|
|
9078
|
+
* `dimensions` is supported on `text-embedding-3-*` only. Asking an older
|
|
9079
|
+
* model for a dimension is an error there, not a silent full-width vector,
|
|
9080
|
+
* which is the failure the framework's own width check would catch anyway.
|
|
9081
|
+
*/
|
|
9082
|
+
/**
|
|
9083
|
+
* An HTTP-shaped failure carrying the status the retry classifier reads — and
|
|
9084
|
+
* the vendor's own error `type`, which the status alone does not distinguish.
|
|
9085
|
+
*
|
|
9086
|
+
* OpenAI answers a spent balance with **429**, the same status as a rate limit:
|
|
9087
|
+
* `{"error":{"type":"insufficient_quota","code":"credit_balance_exhausted"}}`
|
|
9088
|
+
* (observed live, 2026-09-01). One clears by waiting and one never will, so a
|
|
9089
|
+
* classifier reading only the status retries a billing problem five times with
|
|
9090
|
+
* exponential backoff and then reports it — slowly, and as if it had been
|
|
9091
|
+
* transient.
|
|
9092
|
+
*/
|
|
9093
|
+
var OpenAiHttpError = class extends Error {
|
|
9094
|
+
status;
|
|
9095
|
+
/** The vendor's `error.type`, when it sends one. */
|
|
9096
|
+
kind;
|
|
9097
|
+
constructor(status, detail, kind = null) {
|
|
9098
|
+
super(`OpenAI API error ${status}: ${detail}`);
|
|
9099
|
+
this.name = "OpenAiHttpError";
|
|
9100
|
+
this.status = status;
|
|
9101
|
+
this.kind = kind;
|
|
9102
|
+
}
|
|
9103
|
+
};
|
|
9104
|
+
const DEFAULT_BASE = "https://api.openai.com/v1";
|
|
9105
|
+
function openAiRestEmbedClient(apiKey, opts = {}) {
|
|
9106
|
+
return { async embed(params) {
|
|
9107
|
+
const res = await (opts.fetchImpl ?? fetch)(`${opts.baseUrl ?? DEFAULT_BASE}/embeddings`, {
|
|
9108
|
+
method: "POST",
|
|
9109
|
+
headers: {
|
|
9110
|
+
authorization: `Bearer ${apiKey}`,
|
|
9111
|
+
"content-type": "application/json"
|
|
9112
|
+
},
|
|
9113
|
+
body: JSON.stringify({
|
|
9114
|
+
model: params.model,
|
|
9115
|
+
input: [...params.input],
|
|
9116
|
+
dimensions: params.dimensions
|
|
9117
|
+
}),
|
|
9118
|
+
signal: AbortSignal.timeout(params.timeoutMs)
|
|
9119
|
+
});
|
|
9120
|
+
const text = await res.text();
|
|
9121
|
+
if (!res.ok) {
|
|
9122
|
+
let detail = text.slice(0, 300);
|
|
9123
|
+
let kind = null;
|
|
9124
|
+
try {
|
|
9125
|
+
const err = JSON.parse(text).error;
|
|
9126
|
+
if (typeof err?.message === "string") detail = err.message;
|
|
9127
|
+
if (typeof err?.type === "string") kind = err.type;
|
|
9128
|
+
} catch {}
|
|
9129
|
+
throw new OpenAiHttpError(res.status, detail, kind);
|
|
9130
|
+
}
|
|
9131
|
+
const data = [...JSON.parse(text).data ?? []];
|
|
9132
|
+
data.sort((a, b) => (a.index ?? 0) - (b.index ?? 0));
|
|
9133
|
+
return { embeddings: data.map((d) => ({ values: d.embedding })) };
|
|
9134
|
+
} };
|
|
9135
|
+
}
|
|
9136
|
+
/** True for a transport blip with no HTTP status of its own. */
|
|
9137
|
+
function isTransportBlip(exc) {
|
|
9138
|
+
if (exc instanceof OpenAiHttpError) return false;
|
|
9139
|
+
const name = exc?.name;
|
|
9140
|
+
return name === "AbortError" || name === "TimeoutError" || name === "TypeError";
|
|
9141
|
+
}
|
|
9142
|
+
function httpStatusOf(exc) {
|
|
9143
|
+
return exc instanceof OpenAiHttpError ? exc.status : void 0;
|
|
9144
|
+
}
|
|
9145
|
+
/**
|
|
9146
|
+
* The INGEST plane's taxonomy: transport blips, 5xx, AND 429 — batch work is
|
|
9147
|
+
* resumable and has nobody waiting. Deliberately the same shape as the Gemini
|
|
9148
|
+
* adapter's, because the two planes are a property of ksor, not of a vendor.
|
|
9149
|
+
*/
|
|
9150
|
+
function isRetryable(exc) {
|
|
9151
|
+
if (isTransportBlip(exc)) return true;
|
|
9152
|
+
if (exc instanceof OpenAiHttpError && exc.kind === "insufficient_quota") return false;
|
|
9153
|
+
const status = httpStatusOf(exc);
|
|
9154
|
+
if (status === void 0) return false;
|
|
9155
|
+
return status >= 500 && status <= 599 || status === 429;
|
|
9156
|
+
}
|
|
9157
|
+
/**
|
|
9158
|
+
* The READ plane's: transport blips + 5xx only, NEVER 429. A rate-limited
|
|
9159
|
+
* project stays rate-limited on the next second, so a search degrades to
|
|
9160
|
+
* keyword-only now rather than stalling a reader behind backoff.
|
|
9161
|
+
*/
|
|
9162
|
+
function isRetryableQuery(exc) {
|
|
9163
|
+
if (isTransportBlip(exc)) return true;
|
|
9164
|
+
const status = httpStatusOf(exc);
|
|
9165
|
+
return status !== void 0 && status >= 500 && status <= 599;
|
|
9166
|
+
}
|
|
9167
|
+
var OpenAiEmbeddingProvider = class {
|
|
9168
|
+
providerId = "openai";
|
|
9169
|
+
modelId;
|
|
9170
|
+
dim;
|
|
9171
|
+
documentTaskLabel;
|
|
9172
|
+
queryTaskLabel;
|
|
9173
|
+
documentTimeoutMs;
|
|
9174
|
+
queryTimeoutMs;
|
|
9175
|
+
clientFactory;
|
|
9176
|
+
client = null;
|
|
9177
|
+
constructor(opts) {
|
|
9178
|
+
this.modelId = opts.modelId;
|
|
9179
|
+
this.dim = opts.dim;
|
|
9180
|
+
this.documentTaskLabel = opts.documentTaskLabel;
|
|
9181
|
+
this.queryTaskLabel = opts.queryTaskLabel;
|
|
9182
|
+
this.documentTimeoutMs = Math.trunc(opts.documentTimeoutS * 1e3);
|
|
9183
|
+
this.queryTimeoutMs = Math.trunc(opts.queryTimeoutS * 1e3);
|
|
9184
|
+
this.clientFactory = opts.clientFactory ?? (() => openAiRestEmbedClient(opts.apiKey));
|
|
9185
|
+
}
|
|
9186
|
+
get recipe() {
|
|
9187
|
+
return `${this.modelId}/d${this.dim}/${this.documentTaskLabel}`;
|
|
9188
|
+
}
|
|
9189
|
+
getClient() {
|
|
9190
|
+
this.client ??= this.clientFactory();
|
|
9191
|
+
return this.client;
|
|
9192
|
+
}
|
|
9193
|
+
/** DROP the client reference, never close it: in-flight calls finish on their
|
|
9194
|
+
* own reference and the next call rebuilds lazily. Idempotent. */
|
|
9195
|
+
reset() {
|
|
9196
|
+
this.client = null;
|
|
9197
|
+
}
|
|
9198
|
+
async embed(texts, opts) {
|
|
9199
|
+
return (await this.getClient().embed({
|
|
9200
|
+
model: this.modelId,
|
|
9201
|
+
input: texts,
|
|
9202
|
+
dimensions: this.dim,
|
|
9203
|
+
timeoutMs: opts.intent === "document" ? this.documentTimeoutMs : this.queryTimeoutMs
|
|
9204
|
+
})).embeddings.map((e) => [...e.values ?? []]);
|
|
9205
|
+
}
|
|
9052
9206
|
isRetryable(exc) {
|
|
9053
9207
|
return isRetryable(exc);
|
|
9054
9208
|
}
|
|
@@ -9090,11 +9244,30 @@ var MissingProviderKeyError = class extends Error {
|
|
|
9090
9244
|
const PROVIDERS = {
|
|
9091
9245
|
gemini: {
|
|
9092
9246
|
build: (opts) => new GeminiEmbeddingProvider(opts),
|
|
9093
|
-
needsApiKey: true
|
|
9247
|
+
needsApiKey: true,
|
|
9248
|
+
keyEnv: "GEMINI_API_KEY",
|
|
9249
|
+
taskLabels: {
|
|
9250
|
+
document: EMBED_TASK_DOCUMENT,
|
|
9251
|
+
query: EMBED_TASK_QUERY
|
|
9252
|
+
}
|
|
9253
|
+
},
|
|
9254
|
+
openai: {
|
|
9255
|
+
build: (opts) => new OpenAiEmbeddingProvider(opts),
|
|
9256
|
+
needsApiKey: true,
|
|
9257
|
+
keyEnv: "OPENAI_API_KEY",
|
|
9258
|
+
taskLabels: {
|
|
9259
|
+
document: "",
|
|
9260
|
+
query: ""
|
|
9261
|
+
}
|
|
9094
9262
|
},
|
|
9095
9263
|
fake: {
|
|
9096
9264
|
build: (opts) => new FakeEmbeddingProvider(opts),
|
|
9097
|
-
needsApiKey: false
|
|
9265
|
+
needsApiKey: false,
|
|
9266
|
+
keyEnv: null,
|
|
9267
|
+
taskLabels: {
|
|
9268
|
+
document: EMBED_TASK_DOCUMENT,
|
|
9269
|
+
query: EMBED_TASK_QUERY
|
|
9270
|
+
}
|
|
9098
9271
|
}
|
|
9099
9272
|
};
|
|
9100
9273
|
function entryFor(name) {
|
|
@@ -9103,6 +9276,13 @@ function entryFor(name) {
|
|
|
9103
9276
|
return entry;
|
|
9104
9277
|
}
|
|
9105
9278
|
/**
|
|
9279
|
+
* The environment variable this provider's key comes from, or null when it
|
|
9280
|
+
* needs none. Unknown name → the same loud error as building it.
|
|
9281
|
+
*/
|
|
9282
|
+
function providerKeyEnv(name) {
|
|
9283
|
+
return entryFor(name).keyEnv;
|
|
9284
|
+
}
|
|
9285
|
+
/**
|
|
9106
9286
|
* The port door: the named provider bound to the DECLARED embedding space and
|
|
9107
9287
|
* the framework's timeout knobs. `modelId`/`dim` omitted = the shipped
|
|
9108
9288
|
* config space (EMBED_MODEL / EMBED_DIM, eval-locked); an instance may
|
|
@@ -9119,8 +9299,8 @@ function buildShippedProvider(name, opts) {
|
|
|
9119
9299
|
apiKey: opts.apiKey ?? "",
|
|
9120
9300
|
modelId: opts.modelId ?? "gemini-embedding-001",
|
|
9121
9301
|
dim: opts.dim ?? 1536,
|
|
9122
|
-
documentTaskLabel:
|
|
9123
|
-
queryTaskLabel:
|
|
9302
|
+
documentTaskLabel: entry.taskLabels.document,
|
|
9303
|
+
queryTaskLabel: entry.taskLabels.query,
|
|
9124
9304
|
documentTimeoutS: EMBED_TIMEOUT_S(),
|
|
9125
9305
|
queryTimeoutS: QUERY_EMBED_TIMEOUT_S()
|
|
9126
9306
|
});
|
|
@@ -10955,4 +11135,4 @@ function readHandler(ctx) {
|
|
|
10955
11135
|
};
|
|
10956
11136
|
}
|
|
10957
11137
|
//#endregion
|
|
10958
|
-
export { parseViewer as A,
|
|
11138
|
+
export { parseViewer as A, tallyHandlers as B, contentPoolMin as C, outlineHandler as D, keyRingFromEnv as E, recordIsUndescribed as F, z$1 as G, validateViewer as H, runProbe as I, searchHandler as L, prewarmPool as M, providerKeyEnv as N, parseInstanceText as O, readHandler as P, servingPolicy as R, contentPool as S, instancePathOf as T, withPgRetry as U, tlsPosture as V, withProbeDeadline as W, assertGovernanceServable as _, GovernanceGateError as a, checkEmbeddingSpace as b, McpServer$1 as c, READ_ONLY as d, READ_OUTPUT as f, TextSearchConfigMismatch as g, TRUST_TIERS as h, FLOOR as i, pooledEndpointFor as j, parseTrustFloor as k, MissingProviderKeyError as l, SchemaVersionError as m, ContentStoreError as n, MAX_OUTLINE_LIMIT as o, SEARCH_OUTPUT as p, EmbeddingSpaceMismatch as r, MAX_SEARCH_K as s, AudienceError as t, OUTLINE_OUTPUT as u, assertSchemaCompatible as v, embedQueryVlit as w, composeInstructions as x, buildShippedProvider as y, storedTextSearchConfig as z };
|
package/dist/gateway.mjs
CHANGED
|
@@ -1,2 +1,2 @@
|
|
|
1
|
-
import { D as outlineHandler,
|
|
1
|
+
import { D as outlineHandler, F as recordIsUndescribed, G as z$1, L as searchHandler, P as readHandler, c as McpServer$1, d as READ_ONLY, f as READ_OUTPUT, h as TRUST_TIERS, i as FLOOR, o as MAX_OUTLINE_LIMIT, p as SEARCH_OUTPUT, s as MAX_SEARCH_K, u as OUTLINE_OUTPUT, x as composeInstructions } from "./gateway-api-uhx2l1kC-C2BAxISt.mjs";
|
|
2
2
|
export { FLOOR, MAX_OUTLINE_LIMIT, MAX_SEARCH_K, McpServer$1 as McpServer, OUTLINE_OUTPUT, READ_ONLY, READ_OUTPUT, SEARCH_OUTPUT, TRUST_TIERS, composeInstructions, outlineHandler, readHandler, recordIsUndescribed, searchHandler, z$1 as z };
|
package/docs/deploying.md
CHANGED
|
@@ -141,14 +141,38 @@ What the image deliberately does NOT contain (see `.dockerignore`):
|
|
|
141
141
|
> which is the door answering "Method Not Allowed" to a GET rather than a static
|
|
142
142
|
> 404, and is how you tell the door is routed at all.
|
|
143
143
|
>
|
|
144
|
-
> **
|
|
145
|
-
>
|
|
144
|
+
> **Set the Framework Preset to `Services` on the import screen.** Vercel's own
|
|
145
|
+
> guide states it as one of two necessary conditions:
|
|
146
|
+
>
|
|
147
|
+
> > "A project builds as services only when two conditions are both true: the
|
|
148
|
+
> > project's framework is set to Services, and `vercel.json` contains a
|
|
149
|
+
> > `services` key. If either is missing, Vercel falls back to its default
|
|
150
|
+
> > framework detection and ignores your services configuration."
|
|
151
|
+
> > — [vercel.com/kb/guide/vercel-services](https://vercel.com/kb/guide/vercel-services)
|
|
152
|
+
>
|
|
153
|
+
> That is the silent-404 shape exactly: preset `Other` → fallback detection →
|
|
154
|
+
> nothing detected → an empty output that deploys, reports Ready and takes the
|
|
155
|
+
> alias. And no file in your repository can set it for you: `framework` is not
|
|
156
|
+
> a valid top-level key while `services` is present, so this is a project
|
|
157
|
+
> setting or nothing.
|
|
158
|
+
>
|
|
159
|
+
> **One measurement here disagrees with that guide, and is recorded rather than
|
|
160
|
+
> reconciled.** On 2026-08-27 two projects were read back from the Vercel API,
|
|
161
|
+
> one reading preset `Services` and one reading `Other`, and BOTH built the
|
|
162
|
+
> `services` block and served:
|
|
146
163
|
>
|
|
147
164
|
> | project's preset | `services` block built | serves |
|
|
148
165
|
> | ---------------- | ---------------------- | -------------------------------------- |
|
|
149
166
|
> | `Services` | `site` + `door` | `/` 200 · `/llms.txt` 200 · `/mcp` 405 |
|
|
150
167
|
> | `Other` | `site` + `door` | `/` 200 · `/llms.txt` 200 · `/mcp` 405 |
|
|
151
168
|
>
|
|
169
|
+
> Both facts are real and they cannot both be the whole rule. Possible readings
|
|
170
|
+
> — the API's `framework` field being derived rather than the project setting,
|
|
171
|
+
> or Beta behaviour changing between the guide's 2026-08-12 revision and that
|
|
172
|
+
> measurement — are unverified, and guessing between them is what produced the
|
|
173
|
+
> earlier version of this page, which told you the preset was RULED OUT and so
|
|
174
|
+
> steered you away from the one step the vendor calls required. Set the preset.
|
|
175
|
+
>
|
|
152
176
|
> **One failure has been seen that none of this explains.** On a 205-document
|
|
153
177
|
> record (2026-08-26, issue #197) the install ran, `ksor build` ran, every route
|
|
154
178
|
> prerendered — and Vercel collected nothing. The deployment reported **Ready**,
|
|
@@ -427,12 +451,21 @@ Plain `pnpm build` is `[public]`, so the safe thing is the default.
|
|
|
427
451
|
|
|
428
452
|
Two supported answers, and a third that is yours.
|
|
429
453
|
|
|
430
|
-
**
|
|
431
|
-
|
|
432
|
-
`
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
454
|
+
**Run a door per audience.** The door does NOT decide per request: it serves
|
|
455
|
+
ONE viewer list, read from `KSOR_AUDIENCE` once at boot
|
|
456
|
+
(`content-gateway/src/compose.ts`) and validated against the policy before it
|
|
457
|
+
widens past `public`. Every caller holding a valid token for that door gets the
|
|
458
|
+
same view of the record — `docs/authorization.md` says so plainly, and
|
|
459
|
+
per-request visibility filtering is named out of scope in
|
|
460
|
+
`specs/ksor/serve/spec.md`. So the answer is one process per viewer list, each
|
|
461
|
+
with its own `KSOR_AUDIENCE`, behind whatever routing already decides who
|
|
462
|
+
reaches which URL.
|
|
463
|
+
|
|
464
|
+
What the door gives that a static site cannot is the **audit**, not the
|
|
465
|
+
authorization: every read writes a `retrieval_log` row naming the verified
|
|
466
|
+
caller, so "who read what" is answerable afterwards. That is worth having, and
|
|
467
|
+
it is a different thing from "were they allowed to", which is still decided by
|
|
468
|
+
which door they can reach.
|
|
436
469
|
|
|
437
470
|
**Or split the record.** Content needing per-person confidentiality inside one
|
|
438
471
|
tier is usually content that belongs in its own record, with its own gate. That
|
package/docs/ingesting.md
CHANGED
|
@@ -35,12 +35,12 @@ Ingest reads your markdown, sends each new chunk to an embedding provider, and
|
|
|
35
35
|
writes the result to Postgres. So four things must be true, and none of them is
|
|
36
36
|
created for you.
|
|
37
37
|
|
|
38
|
-
| | what
|
|
39
|
-
| -------------------- |
|
|
40
|
-
| **The corpus** | `knowledge/` at your repo root — CommonMark `.md`, one document per file, in the KSoR Profile of OKF: `type`, `title`, `description`, `status` and `ksor.audience` in frontmatter
|
|
41
|
-
| **The database** | Postgres with **pgvector** — `CREATE EXTENSION vector;`
|
|
42
|
-
| **The provider key** | `GEMINI_API_KEY` — the default embedding provider is `gemini-embedding-001`
|
|
43
|
-
| **The DSN** | `KSOR_DB_URL`, named by `instance.md`'s `database.dsn_env`
|
|
38
|
+
| | what | how |
|
|
39
|
+
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
|
|
40
|
+
| **The corpus** | `knowledge/` at your repo root — CommonMark `.md`, one document per file, in the KSoR Profile of OKF: `type`, `title`, `description`, `status` and `ksor.audience` in frontmatter | `pnpm check` validates it and explains any violation; `ksor build` must have written a current `build.lock.json` before ingest will run |
|
|
41
|
+
| **The database** | Postgres with **pgvector** — `CREATE EXTENSION vector;` | any managed host; the DDL below needs a role that can create tables |
|
|
42
|
+
| **The provider key** | `GEMINI_API_KEY` — the default embedding provider is `gemini-embedding-001`. Declaring `embedding.provider: openai` reads `OPENAI_API_KEY` instead: each provider names its own variable | [aistudio.google.com](https://aistudio.google.com/apikey); free tier — embedding input is free of charge |
|
|
43
|
+
| **The DSN** | `KSOR_DB_URL`, named by `instance.md`'s `database.dsn_env` | already named by `instance.md`'s `database:` block |
|
|
44
44
|
|
|
45
45
|
Both variables go in `.env` beside `instance.md` — `ksor` reads it automatically,
|
|
46
46
|
and `.env` is gitignored. Every command below is run **from your repository
|
|
@@ -197,8 +197,25 @@ measure until the corpus is in there.
|
|
|
197
197
|
pnpm exec ksor calibrate --instance instance.md
|
|
198
198
|
```
|
|
199
199
|
|
|
200
|
-
|
|
201
|
-
|
|
200
|
+
**On a free-tier key, use the zero-LLM door instead.** The command above is the
|
|
201
|
+
SYNTHESIZED door: it writes one probe question per sampled passage with an LLM,
|
|
202
|
+
and a free key allows only a few generations a minute — a bigger corpus makes
|
|
203
|
+
that worse, not better. Write your own in-corpus questions, one per line, and
|
|
204
|
+
pass them:
|
|
205
|
+
|
|
206
|
+
```sh
|
|
207
|
+
pnpm exec ksor calibrate --instance instance.md --queries-file questions.txt
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
Six to ten real questions is enough. They should be things this record answers,
|
|
211
|
+
in the words someone would actually ask — the floor is set by the WEAKEST of
|
|
212
|
+
them, so a vague question drags it down and a question the record does not
|
|
213
|
+
answer invalidates the measurement. The door is recorded beside the number
|
|
214
|
+
(`door: queries-file`), because floors from the two doors are measured against
|
|
215
|
+
different distributions and must never be compared as interchangeable.
|
|
216
|
+
|
|
217
|
+
Either way it ends with a block to paste into **`instance.md`**'s frontmatter,
|
|
218
|
+
exactly as printed — the floor, the measurement recorded beside it as a comment, and
|
|
202
219
|
`floor_digest`, the digest of the retrieval predicate the floor was measured
|
|
203
220
|
through. Paste it, then restart `ksor serve`: the floor is read at boot.
|
|
204
221
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@panaversity/ksor",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.54",
|
|
4
4
|
"description": "Knowledge System of Record — compile governed markdown into a static site for people and an MCP server for AI agents, with citations and measured abstention.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"abstention",
|
|
@@ -100,8 +100,11 @@ Stand it up in this order (each step's errors explain how to fix themselves):
|
|
|
100
100
|
```
|
|
101
101
|
|
|
102
102
|
That is enough. `embedding:` is optional and already defaults to
|
|
103
|
-
`provider: gemini`, `model: gemini-embedding-001`, `dim: 1536
|
|
104
|
-
|
|
103
|
+
`provider: gemini`, `model: gemini-embedding-001`, `dim: 1536`. The other
|
|
104
|
+
shipped vendor is OpenAI — `provider: openai`, `model:
|
|
105
|
+
text-embedding-3-small`, `dim: 1536`, key in `OPENAI_API_KEY` — and each
|
|
106
|
+
provider names its own key variable, so nothing else changes. Write the block
|
|
107
|
+
out only to pin the space explicitly or to change it — and note that model and
|
|
105
108
|
dim are the PERSISTED identity of the embedding space, so changing either
|
|
106
109
|
later means re-embedding the whole corpus. Keep `dim` at or below 2000 — the
|
|
107
110
|
schema indexes a `vector` column directly and pgvector's HNSW takes a
|
|
@@ -114,7 +117,33 @@ Stand it up in this order (each step's errors explain how to fix themselves):
|
|
|
114
117
|
Leave `retrieval:` out for now — the gate is off and the server says so.
|
|
115
118
|
Turning it on is step 4, AFTER the record is serving.
|
|
116
119
|
|
|
117
|
-
2. **
|
|
120
|
+
2. **Get the database — your agent can do this one.** `.mcp.json` at the repo
|
|
121
|
+
root declares the MCP servers this project may reach, and the first is Neon.
|
|
122
|
+
With it connected, ask:
|
|
123
|
+
|
|
124
|
+
> Using the Neon MCP server, create a project called `<your-record>` and
|
|
125
|
+
> enable the pgvector extension on it. Then create a branch called `dev`,
|
|
126
|
+
> and save that branch's connection string to `.env` as `KSOR_DB_URL`.
|
|
127
|
+
> Never print my API key. Show me the plan before you run anything.
|
|
128
|
+
|
|
129
|
+
That is `create_project`, `run_sql` (`CREATE EXTENSION vector`),
|
|
130
|
+
`create_branch` and `get_connection_string` — four real tools, no dashboard.
|
|
131
|
+
Prefer the OAuth flow: an API key pasted into `.mcp.json` would be committed,
|
|
132
|
+
because that file carries no secret and is not gitignored.
|
|
133
|
+
|
|
134
|
+
Any Postgres with pgvector works — Neon is the path with an MCP server, not a
|
|
135
|
+
requirement. Locally: `docker run -e POSTGRES_PASSWORD=x -p 5432:5432
|
|
136
|
+
pgvector/pgvector:pg17`.
|
|
137
|
+
|
|
138
|
+
**`GEMINI_API_KEY` is the one step no agent can do.** No vendor mints an API
|
|
139
|
+
key over a protocol; it comes from a browser at
|
|
140
|
+
[aistudio.google.com/apikey](https://aistudio.google.com/apikey) and a human
|
|
141
|
+
pastes it. An agent working through this list should ask for it by name and
|
|
142
|
+
stop, rather than trying — and should say that the FREE TIER is enough:
|
|
143
|
+
embedding input on `gemini-embedding-001` is free of charge, so this is a
|
|
144
|
+
signup, not a bill.
|
|
145
|
+
|
|
146
|
+
3. **Copy `.env.example` to `.env`** and fill it in — `ksor` reads it
|
|
118
147
|
automatically, so nothing needs exporting, and `.env` is already gitignored.
|
|
119
148
|
A real environment variable still wins over the file, so CI and production
|
|
120
149
|
overrides behave normally.
|
|
@@ -134,7 +163,7 @@ Stand it up in this order (each step's errors explain how to fix themselves):
|
|
|
134
163
|
intended dev shape. A PUBLIC deployment configures the SSO door instead —
|
|
135
164
|
see the comments in `.env.example` and "Serving safely" below.
|
|
136
165
|
|
|
137
|
-
|
|
166
|
+
4. **Bring it up.** Once, then every time:
|
|
138
167
|
|
|
139
168
|
```sh
|
|
140
169
|
pnpm provision # schema (or migrate) + grant — the privileged acts, run once
|
|
@@ -219,7 +248,7 @@ Stand it up in this order (each step's errors explain how to fix themselves):
|
|
|
219
248
|
serves, or a DBA who holds the credentials that authorize ingest — not as a
|
|
220
249
|
daily choice.
|
|
221
250
|
|
|
222
|
-
|
|
251
|
+
5. **Turn the abstention gate on — deliberately, once it serves.** This is the
|
|
223
252
|
step that makes "not in this corpus" a real answer, and it is measured, never
|
|
224
253
|
guessed:
|
|
225
254
|
|
|
@@ -171,7 +171,31 @@ That is the whole required config. `embedding:` defaults to Gemini at 1536
|
|
|
171
171
|
dimensions, and `retrieval:` is written for you by step 3's `calibrate`. Change
|
|
172
172
|
the variable name here only if you want a different one.
|
|
173
173
|
|
|
174
|
-
### 2.
|
|
174
|
+
### 2. Get a database — your agent can do this one
|
|
175
|
+
|
|
176
|
+
`.mcp.json` at the repo root declares the MCP servers this project may reach.
|
|
177
|
+
The first is Neon's. With it connected, ask your coding agent:
|
|
178
|
+
|
|
179
|
+
> Using the Neon MCP server, create a project called `<your-record>` and enable
|
|
180
|
+
> the pgvector extension on it. Then create a branch called `dev`, and save that
|
|
181
|
+
> branch's connection string to `.env` as `KSOR_DB_URL`. Never print my API key.
|
|
182
|
+
> Show me the plan before you run anything.
|
|
183
|
+
|
|
184
|
+
Prefer the OAuth flow. `.mcp.json` is committed and carries no secret; an API
|
|
185
|
+
key pasted into it would be.
|
|
186
|
+
|
|
187
|
+
Any Postgres with pgvector works — Neon is the path that has an MCP server, not
|
|
188
|
+
a requirement. Locally:
|
|
189
|
+
`docker run -e POSTGRES_PASSWORD=x -p 5432:5432 pgvector/pgvector:pg17`.
|
|
190
|
+
|
|
191
|
+
**`GEMINI_API_KEY` is the one step no agent can do for you** — no vendor mints
|
|
192
|
+
an API key over a protocol. Get it from
|
|
193
|
+
[aistudio.google.com/apikey](https://aistudio.google.com/apikey) and paste it.
|
|
194
|
+
**The free tier costs nothing and is enough**: embedding input on
|
|
195
|
+
`gemini-embedding-001` is free of charge, and a first corpus is a few thousand
|
|
196
|
+
tokens. This is a signup, not a bill.
|
|
197
|
+
|
|
198
|
+
### 3. Fill in the environment
|
|
175
199
|
|
|
176
200
|
```sh
|
|
177
201
|
cp .env.example .env
|
|
@@ -191,7 +215,7 @@ refusal tells you to _export_ a variable, putting it in `.env` is the same
|
|
|
191
215
|
thing. `KSOR_AUTH=disabled-local` is required for a local run: serve refuses to
|
|
192
216
|
boot unauthenticated on purpose, so a server is never open by accident.
|
|
193
217
|
|
|
194
|
-
###
|
|
218
|
+
### 4. Bring it up
|
|
195
219
|
|
|
196
220
|
```sh
|
|
197
221
|
pnpm provision # once: apply the schema, authorize ingest
|
|
@@ -273,7 +297,15 @@ Delete `.mcp.json`, or keep it — it holds no secret.
|
|
|
273
297
|
Both surfaces on one domain, in about ten minutes:
|
|
274
298
|
|
|
275
299
|
1. **Push the repository to GitHub.**
|
|
276
|
-
2. **Import it in Vercel**, then
|
|
300
|
+
2. **Import it in Vercel**, then set **Framework Preset** to **`Services`** and
|
|
301
|
+
**Root Directory** to `./`. The preset is not cosmetic: Vercel's own guide
|
|
302
|
+
says a project builds as services only when the preset is `Services` AND
|
|
303
|
+
`vercel.json` carries a `services` key, and that "if either is missing,
|
|
304
|
+
Vercel falls back to its default framework detection and ignores your
|
|
305
|
+
services configuration" — which is the silent 404 below. No file in this
|
|
306
|
+
repository can set it for you.
|
|
307
|
+
|
|
308
|
+
As for Root Directory: Vercel
|
|
277
309
|
auto-fills it with `system/site`, because that is where it finds a framework
|
|
278
310
|
— and the build then reads `system/site/vercel.json`, which does not exist,
|
|
279
311
|
and fails with `Project framework is set to "services", but no services are
|
|
@@ -282,6 +314,19 @@ declared`. The services ARE declared, in `vercel.json` at the repo root,
|
|
|
282
314
|
container from the root `Dockerfile`.
|
|
283
315
|
3. **Set three environment variables** in Vercel: `KSOR_DB_URL`,
|
|
284
316
|
`GEMINI_API_KEY`, and `KSOR_AUTH=disabled-public`.
|
|
317
|
+
4. **Check it actually serves**, before you tell anyone the URL. A Ready
|
|
318
|
+
deployment that answers 404 everywhere looks identical to a good one from
|
|
319
|
+
the dashboard:
|
|
320
|
+
|
|
321
|
+
```sh
|
|
322
|
+
B=https://your-record.vercel.app
|
|
323
|
+
curl -o /dev/null -w '%{http_code}\n' "$B/" # expect 200
|
|
324
|
+
curl -o /dev/null -w '%{http_code}\n' "$B/llms.txt" # expect 200
|
|
325
|
+
curl -sI "$B/mcp" | head -1 # expect 405
|
|
326
|
+
```
|
|
327
|
+
|
|
328
|
+
`/mcp` answering 405 is the door refusing a GET — that is how you know it is
|
|
329
|
+
routed at all, and a 404 there means the `services` block was ignored.
|
|
285
330
|
|
|
286
331
|
Three things catch people here. Two are the system being deliberate; the first
|
|
287
332
|
is not, and it is the one that fails without saying so:
|
|
@@ -289,10 +334,13 @@ is not, and it is the one that fails without saying so:
|
|
|
289
334
|
- **A deployment can report Ready and serve nothing.** The build succeeds,
|
|
290
335
|
Vercel collects nothing, and the deployment takes your domain and answers
|
|
291
336
|
`404: NOT_FOUND` everywhere — with one build-log line as the only signal:
|
|
292
|
-
`WARNING! Build output contains no "functions" or "static" directory`.
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
337
|
+
`WARNING! Build output contains no "functions" or "static" directory`. That
|
|
338
|
+
warning is the FALLBACK collector finding nothing, which is what step 2's
|
|
339
|
+
preset exists to prevent — check it first. One measurement of ours disagrees
|
|
340
|
+
with the vendor's rule and is recorded in
|
|
341
|
+
`node_modules/@panaversity/ksor/docs/deploying.md`; the honest state is that
|
|
342
|
+
the preset is necessary by the vendor's documentation and has once appeared
|
|
343
|
+
not to be. If you hit this, the fallback is the
|
|
296
344
|
classic-keys form in `node_modules/@panaversity/ksor/docs/deploying.md` — read
|
|
297
345
|
it there rather than guessing, because it **moves the door off your domain**
|
|
298
346
|
and `KSOR_MCP_RESOURCE_URL` and your SSO API Identifier both have to move with
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
{
|
|
2
|
+
"mcpServers": {
|
|
3
|
+
"Neon": {
|
|
4
|
+
"type": "http",
|
|
5
|
+
"url": "https://mcp.neon.tech/mcp"
|
|
6
|
+
},
|
|
7
|
+
"agentfactory-system-of-record": {
|
|
8
|
+
"type": "http",
|
|
9
|
+
"url": "https://sor.panaversity.org/mcp",
|
|
10
|
+
"oauth": { "clientId": "zia-tutor-ai", "callbackPort": 3118 }
|
|
11
|
+
}
|
|
12
|
+
}
|
|
13
|
+
}
|