@kontextmind/kxm 0.7.55 → 0.7.57
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/CHANGELOG.md +23 -0
- package/docs/configuration.md +1 -0
- package/docs/test-matrix.md +1 -0
- package/package.json +1 -1
- package/plugins/kxm/.claude-plugin/plugin.json +1 -1
- package/plugins/kxm/dist/cli.js +253 -1
- package/plugins/kxm/dist/mcp-server.js +1 -1
- package/plugins/kxm/dist/runtime-supervisor.js +20 -1
- package/plugins/kxm/dist/runtime.js +20 -1
- package/plugins/kxm/package.json +1 -1
- package/plugins/kxm/src/autocomplete.ts +2 -0
- package/plugins/kxm/src/cli/project.ts +74 -3
- package/plugins/kxm/src/cli.ts +8 -0
- package/plugins/kxm/src/hub-env.ts +13 -0
- package/plugins/kxm/src/mcp-server.ts +1 -1
- package/plugins/kxm/src/runtime-supervisor.ts +41 -1
- package/plugins/kxm/src/tenant-status.ts +371 -0
package/CHANGELOG.md
CHANGED
|
@@ -35,6 +35,29 @@ All notable user-facing changes are documented here. The project follows [Semant
|
|
|
35
35
|
|
|
36
36
|
### Changed
|
|
37
37
|
|
|
38
|
+
- **`kxm tenant status`: one composed read for the portal, with the authorities labelled.**
|
|
39
|
+
The portal needs hub metadata (roster, message queue, the hub's run projection) *and* the
|
|
40
|
+
Runtime's authoritative run state, and the failure mode S2 exists to prevent is rendering
|
|
41
|
+
one as the other. The new command reads both and labels every value with its source
|
|
42
|
+
(`hub-projection` vs `runtime-authoritative`); an unreadable upstream becomes
|
|
43
|
+
`unavailable` with a stable reason instead of being filled from the surviving source;
|
|
44
|
+
projection/authoritative disagreements are listed as `discrepancies` rather than averaged;
|
|
45
|
+
and `degraded` marks a partial read so `ok: true` never means "everything was seen". The
|
|
46
|
+
hub read resolves the **admin** credential only — a project token would 401 on the snapshot route —
|
|
47
|
+
and a malformed persisted record degrades that one source as `hub_credential_unreadable` rather than
|
|
48
|
+
aborting the Runtime read with it. Reads run concurrently with a 5 s hub deadline, a 200 with an
|
|
49
|
+
unusable body is `hub_response_invalid` rather than a healthy empty snapshot, and Runtime rows are
|
|
50
|
+
event-log folded (the listing endpoint now folds, so a cached row can no longer be presented as
|
|
51
|
+
state). A run whose fold refuses is labelled `runtime-cached` — the cache, not state — and is
|
|
52
|
+
excluded from the cross-check, which reports `unverifiedFoldRuns` (or `runtime_fold_failed`)
|
|
53
|
+
instead of letting a corrupt event log print "agree"; `kxm runs list` names the same failure on
|
|
54
|
+
the row instead of hiding it. The cross-check is honest about id spaces: hub runs and Runtime
|
|
55
|
+
runs mint ids independently, so zero shared ids reports `unverified` (`run_identity_link_absent`)
|
|
56
|
+
— never agreement — and only cleanly folded shared ids are compared. The Runtime read **attaches** to a live supervisor
|
|
57
|
+
(`attachKxmSupervisor`, new) and never starts one — a portal poller must not conjure a daemon, and
|
|
58
|
+
"nothing is running" is an answer to render, not a condition to repair. Exits non-zero only when
|
|
59
|
+
neither source could be read.
|
|
60
|
+
|
|
38
61
|
- **A Pi producer reply can no longer mint its own success.** `determineOutcome` scanned the
|
|
39
62
|
reply for any declared outcome *word* and, failing that, returned `passed`. So
|
|
40
63
|
`"the gate did not pass, so I would not call this passed"` settled the step as passed — the
|
package/docs/configuration.md
CHANGED
|
@@ -273,6 +273,7 @@ The current hub command groups are `agent`, `session`, `workflow`, `gate`, `hub`
|
|
|
273
273
|
| Command | Purpose |
|
|
274
274
|
|---|---|
|
|
275
275
|
| `kxm init` | Atomically create a provenance-tracked minimal KXM project, validate it without rewriting, resume a pinned interrupted create/repair, apply conflict-free non-authority template updates, or join an existing clone with repeatable `--repository <id=absolute-path>` member bindings stored outside Git. `--dry-run` performs no writes. Provenance-free/ambiguous repair and permission-expanding changes remain planning-only. These configuration slices do **not** activate a KXM Runtime. `kxm init` is project-only; bind a running hub with `kxm hub bind <url>` |
|
|
276
|
+
| `kxm tenant status` | One composed read for machine clients (the portal backend): hub metadata (agent roster, message queue, plans, the hub own workflow runs with stages/progress) and the Runtime **authoritative** run state (event-log folded, per-run `homeRuntimeId`), every value labelled with its source. Reads both sources concurrently with a hub deadline (5 s default); an unreadable upstream becomes `unavailable` with a stable reason (`hub_unreachable`, `hub_timeout`, `hub_unauthorized`, `hub_response_invalid`, `hub_credential_unreadable`, `runtime_supervisor_not_running`) instead of being filled from the other source. The cross-check is honest about id spaces: hub and Runtime runs mint ids independently, so no shared id reports `unverified` (`run_identity_link_absent`) — never agreement; shared ids are compared and disagreements listed; a Runtime row whose event-log fold failed is labelled `runtime-cached` and excluded from the comparison (`unverifiedFoldRuns`), never counted as agreement. `degraded` marks a partial read. Attaches to a live supervisor and never starts one; resolves the **admin** credential only (a project token cannot read the snapshot). Requires a KXM project |
|
|
276
277
|
| `kxm trust diff [--base <rev>]` | Print the structured `kxm.permission-diff.v1` report between a base Git revision (default `HEAD`, materialized into a temporary shadow with a sanitized environment) and the working tree: every authority-bearing field change classified as expansion, narrowing, or neutral with per-field hashes. Performs no project writes |
|
|
277
278
|
| `kxm trust check [--base <rev>]` | Exit non-zero when any expansion exists, so an authority-bearing change cannot merge without a reviewed Git change. Formatting/description-only changes never require review |
|
|
278
279
|
| `kxm run <workflow> [prompt]` | Auto-start the KXM Runtime supervisor if needed, then create an immutable run offline: pins `homeRuntimeId` plus config/executor/tool policy revisions and stores only the prompt hash. `--dry-run` prints the plan without creating anything |
|
package/docs/test-matrix.md
CHANGED
|
@@ -13,6 +13,7 @@ npm run verify
|
|
|
13
13
|
|
|
14
14
|
| Feature | Automated evidence |
|
|
15
15
|
|---|---|
|
|
16
|
+
| Portal tenant read composes two labelled authorities: hub runs carry `hub-projection`, Runtime runs carry `runtime-authoritative`; a down hub leaves runtime state readable and vice versa with stable reasons; projection/authoritative disagreements surface as `discrepancies`; binding scope is labelled loopback/remote | `test/core/studio-layout.test.ts` |
|
|
16
17
|
| Structured-result settlement only: outcome words in prose, empty replies, and outcomes outside the step's declared set all fail closed; a declared JSON object and a JSON result block inside prose both settle, and the engine records `outcome_unknown` and terminates `failed` | `test/core/pi-producer.test.ts`, `test/core/engine.test.ts` |
|
|
17
18
|
| Health, readiness, metrics, request IDs, security headers | `test/core/hub-api.test.ts` |
|
|
18
19
|
| Shared and per-project authentication, project isolation | `test/core/hub-api.test.ts` |
|
package/package.json
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
|
|
3
3
|
"name": "kxm",
|
|
4
4
|
"displayName": "KXM",
|
|
5
|
-
"version": "0.7.
|
|
5
|
+
"version": "0.7.57",
|
|
6
6
|
"description": "Headless multi-agent orchestration, durable workflows, and a live operator dashboard for Pi and Claude Code",
|
|
7
7
|
"author": {
|
|
8
8
|
"name": "KontextMind",
|
package/plugins/kxm/dist/cli.js
CHANGED
|
@@ -19678,6 +19678,11 @@ function resolveClientHubAuthToken(env, project) {
|
|
|
19678
19678
|
const record = readHubEnvRecord(env);
|
|
19679
19679
|
return record?.projectTokens?.[project]?.trim() || record?.authToken?.trim() || void 0;
|
|
19680
19680
|
}
|
|
19681
|
+
function resolveClientAdminAuthToken(env = process.env) {
|
|
19682
|
+
const envToken = env.KXM_AUTH_TOKEN?.trim();
|
|
19683
|
+
if (envToken) return envToken;
|
|
19684
|
+
return readHubEnvRecord(env)?.authToken?.trim() || void 0;
|
|
19685
|
+
}
|
|
19681
19686
|
|
|
19682
19687
|
// plugins/kxm/src/project-name.ts
|
|
19683
19688
|
import { readFileSync as readFileSync3 } from "node:fs";
|
|
@@ -25725,6 +25730,15 @@ async function probeSupervisor(port, expectedRuntimeId, token, timeoutMs = 750)
|
|
|
25725
25730
|
function hashKxmTokenProof(token, nonce) {
|
|
25726
25731
|
return createHmac("sha256", token).update(`kxm-runtime-token-proof\0${nonce}`, "utf8").digest("hex");
|
|
25727
25732
|
}
|
|
25733
|
+
async function attachKxmSupervisor(options = {}) {
|
|
25734
|
+
const paths = kxmRuntimePaths(options.stateRoot !== void 0 ? { stateRoot: options.stateRoot } : { ...options.env ? { env: options.env } : {} });
|
|
25735
|
+
const status = kxmSupervisorStatus(paths);
|
|
25736
|
+
if (!status.running || !status.port || !status.runtimeId) return void 0;
|
|
25737
|
+
const token = readKxmSupervisorToken(paths);
|
|
25738
|
+
if (!token) return void 0;
|
|
25739
|
+
if (!await probeSupervisor(status.port, status.runtimeId, token)) return void 0;
|
|
25740
|
+
return { runtimeId: status.runtimeId, port: status.port, token, started: false };
|
|
25741
|
+
}
|
|
25728
25742
|
async function ensureKxmSupervisor(options = {}) {
|
|
25729
25743
|
const paths = kxmRuntimePaths(options.stateRoot !== void 0 ? { stateRoot: options.stateRoot } : { ...options.env ? { env: options.env } : {} });
|
|
25730
25744
|
const status = kxmSupervisorStatus(paths);
|
|
@@ -31761,6 +31775,176 @@ function initializeKxmProject(start = process.cwd(), options = {}) {
|
|
|
31761
31775
|
return withKxmLocalBindingLock(gitRoot, storeOptions, (lock) => initializeKxmProjectAtGitRoot(start, gitRoot, options, lock));
|
|
31762
31776
|
}
|
|
31763
31777
|
|
|
31778
|
+
// plugins/kxm/src/tenant-status.ts
|
|
31779
|
+
var TENANT_STATUS_SCHEMA = "kxm.tenant-status.v1";
|
|
31780
|
+
var DEFAULT_HUB_TIMEOUT_MS = 5e3;
|
|
31781
|
+
function hubSourceReason(status) {
|
|
31782
|
+
if (status === 401 || status === 403) return "hub_unauthorized";
|
|
31783
|
+
if (status === 404) return "hub_not_found";
|
|
31784
|
+
return `hub_http_${status}`;
|
|
31785
|
+
}
|
|
31786
|
+
function asString(value) {
|
|
31787
|
+
return typeof value === "string" && value.length > 0 ? value : void 0;
|
|
31788
|
+
}
|
|
31789
|
+
function isObject(value) {
|
|
31790
|
+
return typeof value === "object" && value !== null && !Array.isArray(value);
|
|
31791
|
+
}
|
|
31792
|
+
async function assembleTenantStatus(input) {
|
|
31793
|
+
const now = input.now ?? (() => /* @__PURE__ */ new Date());
|
|
31794
|
+
const base = input.hubUrl.replace(/\/$/, "");
|
|
31795
|
+
const at = () => now().toISOString();
|
|
31796
|
+
const readHub = async () => {
|
|
31797
|
+
const attemptedAt = at();
|
|
31798
|
+
let token;
|
|
31799
|
+
try {
|
|
31800
|
+
token = input.resolveAdminToken();
|
|
31801
|
+
} catch (error) {
|
|
31802
|
+
void error;
|
|
31803
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: "hub_credential_unreadable" };
|
|
31804
|
+
}
|
|
31805
|
+
try {
|
|
31806
|
+
const response = await input.fetchImpl(`${base}/v1/ops/snapshot?project=${encodeURIComponent(input.project)}`, {
|
|
31807
|
+
headers: token ? { authorization: `Bearer ${token}` } : {},
|
|
31808
|
+
signal: AbortSignal.timeout(input.hubTimeoutMs ?? DEFAULT_HUB_TIMEOUT_MS)
|
|
31809
|
+
});
|
|
31810
|
+
if (!response.ok) {
|
|
31811
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: hubSourceReason(response.status) };
|
|
31812
|
+
}
|
|
31813
|
+
let body;
|
|
31814
|
+
try {
|
|
31815
|
+
body = await response.json();
|
|
31816
|
+
} catch (error) {
|
|
31817
|
+
const name = error instanceof Error ? error.name : void 0;
|
|
31818
|
+
if (name === "TimeoutError" || name === "AbortError") {
|
|
31819
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: "hub_timeout" };
|
|
31820
|
+
}
|
|
31821
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: "hub_response_invalid" };
|
|
31822
|
+
}
|
|
31823
|
+
if (!isObject(body) || typeof body.project !== "string" || typeof body.fetchedAt !== "string" || !Array.isArray(body.agents) || !Array.isArray(body.runs) || !Array.isArray(body.plans ?? []) || typeof body.openMessageTotal !== "number" || typeof body.runTotal !== "number") {
|
|
31824
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: "hub_response_invalid" };
|
|
31825
|
+
}
|
|
31826
|
+
if (body.project !== input.project) {
|
|
31827
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: "hub_project_mismatch" };
|
|
31828
|
+
}
|
|
31829
|
+
const runRows = body.runs.filter(isObject);
|
|
31830
|
+
const value = {
|
|
31831
|
+
project: body.project,
|
|
31832
|
+
fetchedAt: body.fetchedAt,
|
|
31833
|
+
agents: body.agents.filter(isObject).map((agent) => ({
|
|
31834
|
+
id: asString(agent.id) ?? "",
|
|
31835
|
+
name: asString(agent.name) ?? "",
|
|
31836
|
+
online: agent.online === true
|
|
31837
|
+
})),
|
|
31838
|
+
openMessageTotal: body.openMessageTotal,
|
|
31839
|
+
runTotal: body.runTotal,
|
|
31840
|
+
runs: runRows.map((run) => {
|
|
31841
|
+
const stages = Array.isArray(run.stages) ? run.stages.filter(isObject).map((stage) => ({
|
|
31842
|
+
id: asString(stage.id) ?? "",
|
|
31843
|
+
status: asString(stage.status) ?? "unknown",
|
|
31844
|
+
...typeof stage.attempts === "number" ? { attempts: stage.attempts } : {}
|
|
31845
|
+
})) : void 0;
|
|
31846
|
+
const done = stages?.filter((stage) => stage.status === "passed" || stage.status === "failed" || stage.status === "warning").length;
|
|
31847
|
+
return {
|
|
31848
|
+
id: asString(run.id) ?? "",
|
|
31849
|
+
status: asString(run.status) ?? "unknown",
|
|
31850
|
+
definitionId: asString(run.definitionId) ?? "",
|
|
31851
|
+
...asString(run.currentStage) ? { currentStage: asString(run.currentStage) } : {},
|
|
31852
|
+
...asString(run.targetAgentName) ? { targetAgentName: asString(run.targetAgentName) } : {},
|
|
31853
|
+
...asString(run.updatedAt) ? { updatedAt: asString(run.updatedAt) } : {},
|
|
31854
|
+
...stages !== void 0 ? { progress: { done: done ?? 0, total: stages.length }, stages } : {},
|
|
31855
|
+
source: "hub-projection"
|
|
31856
|
+
};
|
|
31857
|
+
}),
|
|
31858
|
+
plans: (Array.isArray(body.plans) ? body.plans : []).filter(isObject).map((plan) => ({
|
|
31859
|
+
id: asString(plan.id) ?? "",
|
|
31860
|
+
runId: asString(plan.runId) ?? "",
|
|
31861
|
+
summary: asString(plan.summary) ?? "",
|
|
31862
|
+
createdAt: asString(plan.createdAt) ?? "",
|
|
31863
|
+
...asString(plan.stageId) ? { stageId: asString(plan.stageId) } : {},
|
|
31864
|
+
...asString(plan.severity) ? { severity: asString(plan.severity) } : {}
|
|
31865
|
+
}))
|
|
31866
|
+
};
|
|
31867
|
+
return { state: "ok", observedAt: attemptedAt, value };
|
|
31868
|
+
} catch (error) {
|
|
31869
|
+
const name = error instanceof Error ? error.name : void 0;
|
|
31870
|
+
if (name === "TimeoutError" || name === "AbortError") {
|
|
31871
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: "hub_timeout" };
|
|
31872
|
+
}
|
|
31873
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: "hub_unreachable" };
|
|
31874
|
+
}
|
|
31875
|
+
};
|
|
31876
|
+
const readRuntime = async () => {
|
|
31877
|
+
const attemptedAt = at();
|
|
31878
|
+
try {
|
|
31879
|
+
const runs = await input.runtime.listRuns();
|
|
31880
|
+
return { state: "ok", observedAt: attemptedAt, value: { runs } };
|
|
31881
|
+
} catch (error) {
|
|
31882
|
+
const code = error instanceof Error && "issues" in error && Array.isArray(error.issues) && typeof error.issues[0]?.code === "string" ? error.issues[0].code : void 0;
|
|
31883
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: code ?? "runtime_unavailable" };
|
|
31884
|
+
}
|
|
31885
|
+
};
|
|
31886
|
+
const [hub, runtime] = await Promise.all([readHub(), readRuntime()]);
|
|
31887
|
+
let runComparison;
|
|
31888
|
+
if (!hub.value || !runtime.value) {
|
|
31889
|
+
runComparison = {
|
|
31890
|
+
state: "unavailable",
|
|
31891
|
+
reason: !hub.value ? `hub_${hub.reason ?? "unavailable"}` : `runtime_${runtime.reason ?? "unavailable"}`
|
|
31892
|
+
};
|
|
31893
|
+
} else {
|
|
31894
|
+
const authoritative = new Map(
|
|
31895
|
+
runtime.value.runs.filter((run) => run.projectionError === void 0).map((run) => [run.runId, run.status])
|
|
31896
|
+
);
|
|
31897
|
+
const hubIds = new Set(hub.value.runs.map((run) => run.id));
|
|
31898
|
+
const foldFailed = runtime.value.runs.filter((run) => run.projectionError !== void 0 && hubIds.has(run.runId));
|
|
31899
|
+
const discrepancies = [];
|
|
31900
|
+
let matched = 0;
|
|
31901
|
+
for (const projected of hub.value.runs) {
|
|
31902
|
+
const runtimeStatus = authoritative.get(projected.id);
|
|
31903
|
+
if (runtimeStatus === void 0) continue;
|
|
31904
|
+
matched += 1;
|
|
31905
|
+
if (runtimeStatus !== projected.status) {
|
|
31906
|
+
discrepancies.push({ runId: projected.id, hubStatus: projected.status, runtimeStatus });
|
|
31907
|
+
}
|
|
31908
|
+
}
|
|
31909
|
+
if (matched > 0) {
|
|
31910
|
+
runComparison = {
|
|
31911
|
+
state: "compared",
|
|
31912
|
+
matched,
|
|
31913
|
+
...foldFailed.length > 0 ? { unverifiedFoldRuns: foldFailed.length } : {},
|
|
31914
|
+
...discrepancies.length > 0 ? { discrepancies } : {}
|
|
31915
|
+
};
|
|
31916
|
+
} else {
|
|
31917
|
+
runComparison = foldFailed.length > 0 ? { state: "unverified", reason: "runtime_fold_failed", matched: 0, unverifiedFoldRuns: foldFailed.length } : { state: "unverified", reason: "run_identity_link_absent", matched: 0 };
|
|
31918
|
+
}
|
|
31919
|
+
}
|
|
31920
|
+
return {
|
|
31921
|
+
schema: TENANT_STATUS_SCHEMA,
|
|
31922
|
+
project: input.project,
|
|
31923
|
+
generatedAt: at(),
|
|
31924
|
+
hubUrl: input.hubUrl,
|
|
31925
|
+
bindingScope: hubBindingScope(input.hubUrl),
|
|
31926
|
+
hub,
|
|
31927
|
+
runtime,
|
|
31928
|
+
runComparison,
|
|
31929
|
+
degraded: hub.state !== "ok" || runtime.state !== "ok"
|
|
31930
|
+
};
|
|
31931
|
+
}
|
|
31932
|
+
function formatTenantStatus(payload) {
|
|
31933
|
+
const hubLine = payload.hub.state === "ok" && payload.hub.value ? `hub ${payload.hub.value.agents.filter((agent) => agent.online).length}/${payload.hub.value.agents.length} agents online, ${payload.hub.value.runs.length} hub runs, ${payload.hub.value.openMessageTotal} open messages` : `hub unavailable (${payload.hub.reason ?? "unknown"})`;
|
|
31934
|
+
const runtimeLine = payload.runtime.state === "ok" && payload.runtime.value ? (() => {
|
|
31935
|
+
const cached = payload.runtime.value.runs.filter((run) => run.projectionError !== void 0).length;
|
|
31936
|
+
const authoritative = payload.runtime.value.runs.length - cached;
|
|
31937
|
+
return `runtime ${authoritative} runs (authoritative, on this box)${cached > 0 ? `, ${cached} cached (fold failed, not state)` : ""}`;
|
|
31938
|
+
})() : `runtime unavailable (${payload.runtime.reason ?? "unknown"})`;
|
|
31939
|
+
const comparisonLine = payload.runComparison.state === "compared" ? payload.runComparison.discrepancies && payload.runComparison.discrepancies.length > 0 ? `cross-check: ${payload.runComparison.discrepancies.length} of ${payload.runComparison.matched} matched run(s) disagree${payload.runComparison.unverifiedFoldRuns ? ` (${payload.runComparison.unverifiedFoldRuns} unverified: fold failed)` : ""}` : `cross-check: ${payload.runComparison.matched} matched run(s) agree${payload.runComparison.unverifiedFoldRuns ? ` (${payload.runComparison.unverifiedFoldRuns} unverified: fold failed)` : ""}` : payload.runComparison.state === "unverified" ? payload.runComparison.reason === "runtime_fold_failed" ? `cross-check: unavailable \u2014 ${payload.runComparison.unverifiedFoldRuns ?? 0} shared run(s) could not be verified (runtime fold failed)` : "cross-check: unavailable \u2014 no run id appears in both sources (independent id spaces)" : `cross-check: unavailable (${payload.runComparison.reason ?? "unknown"})`;
|
|
31940
|
+
return [
|
|
31941
|
+
`tenant ${payload.project} @ ${payload.hubUrl} (${payload.bindingScope})`,
|
|
31942
|
+
hubLine,
|
|
31943
|
+
runtimeLine,
|
|
31944
|
+
comparisonLine
|
|
31945
|
+
].join("\n");
|
|
31946
|
+
}
|
|
31947
|
+
|
|
31764
31948
|
// plugins/kxm/src/cli/project.ts
|
|
31765
31949
|
var kxmDriveCliSeams = {};
|
|
31766
31950
|
function initPlanPayload(plan) {
|
|
@@ -32236,7 +32420,7 @@ async function cmdKxmRunList(runtime) {
|
|
|
32236
32420
|
runtime.io,
|
|
32237
32421
|
runtime.json,
|
|
32238
32422
|
{ ok: true, command: "runs list", runs },
|
|
32239
|
-
runs.length === 0 ? "no runs" : runs.map((run) => `${run.runId} ${run.status} ${run.workflowId} ${run.createdAt}`).join("\n")
|
|
32423
|
+
runs.length === 0 ? "no runs" : runs.map((run) => `${run.runId} ${run.status} ${run.workflowId} ${run.createdAt}${run.projectionError ? ` [state unverified: ${run.projectionError}]` : ""}`).join("\n")
|
|
32240
32424
|
);
|
|
32241
32425
|
return 0;
|
|
32242
32426
|
} catch (error) {
|
|
@@ -32248,6 +32432,67 @@ async function cmdKxmRunList(runtime) {
|
|
|
32248
32432
|
return 1;
|
|
32249
32433
|
}
|
|
32250
32434
|
}
|
|
32435
|
+
async function cmdTenantStatus(runtime) {
|
|
32436
|
+
let projectRoot;
|
|
32437
|
+
let projectId;
|
|
32438
|
+
try {
|
|
32439
|
+
projectRoot = discoverKxmProjectRoot(runtime.cwd) ?? void 0;
|
|
32440
|
+
if (!projectRoot) {
|
|
32441
|
+
print(runtime.io, runtime.json, { ok: false, command: "tenant status", error: "project_required" }, "kxm tenant status requires a KXM project (run kxm init first)");
|
|
32442
|
+
return 1;
|
|
32443
|
+
}
|
|
32444
|
+
const bundle = loadKxmProject(projectRoot, {});
|
|
32445
|
+
projectId = String(bundle.project.value.id);
|
|
32446
|
+
const root = projectRoot;
|
|
32447
|
+
const payload = await assembleTenantStatus({
|
|
32448
|
+
project: projectId,
|
|
32449
|
+
hubUrl: runtime.serverUrl,
|
|
32450
|
+
// Admin-scoped route: resolve the admin credential only (a project token would 401),
|
|
32451
|
+
// and resolve it inside the hub source so a malformed persisted record degrades that
|
|
32452
|
+
// one source instead of aborting the Runtime read with it.
|
|
32453
|
+
resolveAdminToken: () => resolveClientAdminAuthToken(runtime.env),
|
|
32454
|
+
fetchImpl: runtime.fetchImpl,
|
|
32455
|
+
runtime: {
|
|
32456
|
+
listRuns: async () => {
|
|
32457
|
+
const handle = await attachKxmSupervisor({ env: runtime.env });
|
|
32458
|
+
if (!handle) {
|
|
32459
|
+
throw runtimeError("runtime_supervisor_not_running", "runtime", "no live runtime supervisor on this box");
|
|
32460
|
+
}
|
|
32461
|
+
const result = await kxmRuntimeRequest(handle, "GET", `/v1/projects/${encodeURIComponent(projectId)}/runs?projectRoot=${encodeURIComponent(root)}`);
|
|
32462
|
+
const runs = result.runs ?? [];
|
|
32463
|
+
return runs.map((run) => ({
|
|
32464
|
+
runId: String(run.runId ?? ""),
|
|
32465
|
+
status: String(run.status ?? "unknown"),
|
|
32466
|
+
homeRuntimeId: String(run.homeRuntimeId ?? ""),
|
|
32467
|
+
...typeof run.workflowId === "string" ? { workflowId: run.workflowId } : {},
|
|
32468
|
+
...typeof run.createdAt === "string" ? { createdAt: run.createdAt } : {},
|
|
32469
|
+
...typeof run.updatedAt === "string" ? { updatedAt: run.updatedAt } : {},
|
|
32470
|
+
...typeof run.projectionError === "string" ? { projectionError: run.projectionError } : {},
|
|
32471
|
+
source: typeof run.projectionError === "string" ? "runtime-cached" : "runtime-authoritative"
|
|
32472
|
+
}));
|
|
32473
|
+
}
|
|
32474
|
+
}
|
|
32475
|
+
});
|
|
32476
|
+
if (payload.hub.state !== "ok" && payload.runtime.state !== "ok") {
|
|
32477
|
+
print(
|
|
32478
|
+
runtime.io,
|
|
32479
|
+
runtime.json,
|
|
32480
|
+
{ ok: false, command: "tenant status", error: "tenant_status_no_source", payload },
|
|
32481
|
+
`neither source could be read: hub ${payload.hub.reason ?? "unknown"}, runtime ${payload.runtime.reason ?? "unknown"}`
|
|
32482
|
+
);
|
|
32483
|
+
return 1;
|
|
32484
|
+
}
|
|
32485
|
+
print(runtime.io, runtime.json, { ok: true, command: "tenant status", ...payload }, formatTenantStatus(payload));
|
|
32486
|
+
return 0;
|
|
32487
|
+
} catch (error) {
|
|
32488
|
+
if (error instanceof KxmConfigError) {
|
|
32489
|
+
print(runtime.io, runtime.json, { ok: false, command: "tenant status", error: "tenant_status_failed", issues: error.issues }, `tenant status failed: ${error.message}`);
|
|
32490
|
+
return 1;
|
|
32491
|
+
}
|
|
32492
|
+
print(runtime.io, runtime.json, { ok: false, command: "tenant status", error: "tenant_status_io_failed" }, "tenant status failed because a local operation did not complete");
|
|
32493
|
+
return 1;
|
|
32494
|
+
}
|
|
32495
|
+
}
|
|
32251
32496
|
async function cmdHarnessList(runtime) {
|
|
32252
32497
|
const inventory = await probeHarnessesAsync({ env: runtime.env });
|
|
32253
32498
|
print(runtime.io, runtime.json, { ok: true, command: "harness list", ...inventory }, formatHarnessInventory(inventory));
|
|
@@ -45628,6 +45873,7 @@ var TOP_LEVEL_COMMANDS = [
|
|
|
45628
45873
|
"restore",
|
|
45629
45874
|
"run",
|
|
45630
45875
|
"runs",
|
|
45876
|
+
"tenant",
|
|
45631
45877
|
"harness",
|
|
45632
45878
|
"update",
|
|
45633
45879
|
"runtime",
|
|
@@ -45656,6 +45902,7 @@ var TOP_LEVEL_COMMANDS = [
|
|
|
45656
45902
|
];
|
|
45657
45903
|
var SUBCOMMANDS = {
|
|
45658
45904
|
runs: ["status", "cancel", "list"],
|
|
45905
|
+
tenant: ["status"],
|
|
45659
45906
|
harness: ["list"],
|
|
45660
45907
|
runtime: ["start", "status", "stop"],
|
|
45661
45908
|
trust: ["diff", "check"],
|
|
@@ -47372,6 +47619,11 @@ function createProgram(ctx, result) {
|
|
|
47372
47619
|
addGlobalOptions(runCmd.command("list").description("List recent runs for the current project")).action(async function runListAction() {
|
|
47373
47620
|
result.code = await cmdKxmRunList(runtimeFrom(ctx, this));
|
|
47374
47621
|
});
|
|
47622
|
+
const tenantCmd = addGlobalOptions(program2.command("tenant").description("Composed tenant reads for machine clients (portal)"));
|
|
47623
|
+
tenantCmd.helpCommand("help", "Show tenant help");
|
|
47624
|
+
addGlobalOptions(tenantCmd.command("status").description("Read hub metadata and authoritative Runtime run state as one labeled view")).action(async function tenantStatusAction() {
|
|
47625
|
+
result.code = await cmdTenantStatus(runtimeFrom(ctx, this));
|
|
47626
|
+
});
|
|
47375
47627
|
const modelsCmd = addGlobalOptions(program2.command("models").description("Manage model catalogs, roles, and route state"));
|
|
47376
47628
|
modelsCmd.action(async function modelsScreenAction() {
|
|
47377
47629
|
result.code = await cmdModelsScreen(runtimeFrom(ctx, this));
|
|
@@ -17121,7 +17121,7 @@ async function deliverInboxNotification(messageId, delivered, notify) {
|
|
|
17121
17121
|
}
|
|
17122
17122
|
|
|
17123
17123
|
// plugins/kxm/src/mcp-server.ts
|
|
17124
|
-
var VERSION = "0.7.
|
|
17124
|
+
var VERSION = "0.7.57";
|
|
17125
17125
|
var inbox = /* @__PURE__ */ new Map();
|
|
17126
17126
|
var notifiedInbox = /* @__PURE__ */ new Set();
|
|
17127
17127
|
var meshClient;
|
|
@@ -26837,6 +26837,15 @@ async function probeSupervisor(port, expectedRuntimeId, token, timeoutMs = 750)
|
|
|
26837
26837
|
function hashKxmTokenProof(token, nonce) {
|
|
26838
26838
|
return createHmac("sha256", token).update(`kxm-runtime-token-proof\0${nonce}`, "utf8").digest("hex");
|
|
26839
26839
|
}
|
|
26840
|
+
async function attachKxmSupervisor(options = {}) {
|
|
26841
|
+
const paths = kxmRuntimePaths(options.stateRoot !== void 0 ? { stateRoot: options.stateRoot } : { ...options.env ? { env: options.env } : {} });
|
|
26842
|
+
const status = kxmSupervisorStatus(paths);
|
|
26843
|
+
if (!status.running || !status.port || !status.runtimeId) return void 0;
|
|
26844
|
+
const token = readKxmSupervisorToken(paths);
|
|
26845
|
+
if (!token) return void 0;
|
|
26846
|
+
if (!await probeSupervisor(status.port, status.runtimeId, token)) return void 0;
|
|
26847
|
+
return { runtimeId: status.runtimeId, port: status.port, token, started: false };
|
|
26848
|
+
}
|
|
26840
26849
|
async function ensureKxmSupervisor(options = {}) {
|
|
26841
26850
|
const paths = kxmRuntimePaths(options.stateRoot !== void 0 ? { stateRoot: options.stateRoot } : { ...options.env ? { env: options.env } : {} });
|
|
26842
26851
|
const status = kxmSupervisorStatus(paths);
|
|
@@ -27246,7 +27255,16 @@ async function startKxmRuntimeSupervisorInner(paths, requestedPortOption, now) {
|
|
|
27246
27255
|
sendJson(response, 400, { ok: false, error: "runtime_request_invalid", message: `project ${requestedProjectId} is not the bound project ${context.projectId}` });
|
|
27247
27256
|
return;
|
|
27248
27257
|
}
|
|
27249
|
-
const runs = context.eventStore.runsForProject(requestedProjectId, 50)
|
|
27258
|
+
const runs = context.eventStore.runsForProject(requestedProjectId, 50).map((stored) => {
|
|
27259
|
+
try {
|
|
27260
|
+
return projectKxmRunReadOnly(context, stored.runId);
|
|
27261
|
+
} catch (error) {
|
|
27262
|
+
return {
|
|
27263
|
+
...stored,
|
|
27264
|
+
projectionError: error instanceof KxmConfigError ? error.issues[0]?.code ?? "runtime_projection_failed" : "runtime_projection_failed"
|
|
27265
|
+
};
|
|
27266
|
+
}
|
|
27267
|
+
});
|
|
27250
27268
|
sendJson(response, 200, { ok: true, runs });
|
|
27251
27269
|
return;
|
|
27252
27270
|
}
|
|
@@ -27379,6 +27397,7 @@ async function kxmRuntimeRequest(handle, method, path, body) {
|
|
|
27379
27397
|
export {
|
|
27380
27398
|
DEFAULT_RUNTIME_STOP_GRACE_MS,
|
|
27381
27399
|
MAX_RUNTIME_STOP_GRACE_MS,
|
|
27400
|
+
attachKxmSupervisor,
|
|
27382
27401
|
ensureKxmSupervisor,
|
|
27383
27402
|
hashKxmSupervisorToken,
|
|
27384
27403
|
hashKxmTokenProof,
|
|
@@ -27349,6 +27349,15 @@ async function probeSupervisor(port, expectedRuntimeId, token, timeoutMs = 750)
|
|
|
27349
27349
|
function hashKxmTokenProof(token, nonce) {
|
|
27350
27350
|
return createHmac("sha256", token).update(`kxm-runtime-token-proof\0${nonce}`, "utf8").digest("hex");
|
|
27351
27351
|
}
|
|
27352
|
+
async function attachKxmSupervisor(options = {}) {
|
|
27353
|
+
const paths = kxmRuntimePaths(options.stateRoot !== void 0 ? { stateRoot: options.stateRoot } : { ...options.env ? { env: options.env } : {} });
|
|
27354
|
+
const status = kxmSupervisorStatus(paths);
|
|
27355
|
+
if (!status.running || !status.port || !status.runtimeId) return void 0;
|
|
27356
|
+
const token = readKxmSupervisorToken(paths);
|
|
27357
|
+
if (!token) return void 0;
|
|
27358
|
+
if (!await probeSupervisor(status.port, status.runtimeId, token)) return void 0;
|
|
27359
|
+
return { runtimeId: status.runtimeId, port: status.port, token, started: false };
|
|
27360
|
+
}
|
|
27352
27361
|
async function ensureKxmSupervisor(options = {}) {
|
|
27353
27362
|
const paths = kxmRuntimePaths(options.stateRoot !== void 0 ? { stateRoot: options.stateRoot } : { ...options.env ? { env: options.env } : {} });
|
|
27354
27363
|
const status = kxmSupervisorStatus(paths);
|
|
@@ -27758,7 +27767,16 @@ async function startKxmRuntimeSupervisorInner(paths, requestedPortOption, now) {
|
|
|
27758
27767
|
sendJson(response, 400, { ok: false, error: "runtime_request_invalid", message: `project ${requestedProjectId} is not the bound project ${context.projectId}` });
|
|
27759
27768
|
return;
|
|
27760
27769
|
}
|
|
27761
|
-
const runs = context.eventStore.runsForProject(requestedProjectId, 50)
|
|
27770
|
+
const runs = context.eventStore.runsForProject(requestedProjectId, 50).map((stored) => {
|
|
27771
|
+
try {
|
|
27772
|
+
return projectKxmRunReadOnly(context, stored.runId);
|
|
27773
|
+
} catch (error) {
|
|
27774
|
+
return {
|
|
27775
|
+
...stored,
|
|
27776
|
+
projectionError: error instanceof KxmConfigError ? error.issues[0]?.code ?? "runtime_projection_failed" : "runtime_projection_failed"
|
|
27777
|
+
};
|
|
27778
|
+
}
|
|
27779
|
+
});
|
|
27762
27780
|
sendJson(response, 200, { ok: true, runs });
|
|
27763
27781
|
return;
|
|
27764
27782
|
}
|
|
@@ -30227,6 +30245,7 @@ export {
|
|
|
30227
30245
|
acceptKxmRun,
|
|
30228
30246
|
assertClosedGateObservation,
|
|
30229
30247
|
assertKxmConfigError,
|
|
30248
|
+
attachKxmSupervisor,
|
|
30230
30249
|
backupDatabaseFile,
|
|
30231
30250
|
buildImprovementReport,
|
|
30232
30251
|
buildSshArgs,
|
package/plugins/kxm/package.json
CHANGED
|
@@ -6,6 +6,7 @@ const TOP_LEVEL_COMMANDS = [
|
|
|
6
6
|
"restore",
|
|
7
7
|
"run",
|
|
8
8
|
"runs",
|
|
9
|
+
"tenant",
|
|
9
10
|
"harness",
|
|
10
11
|
"update",
|
|
11
12
|
"runtime",
|
|
@@ -35,6 +36,7 @@ const TOP_LEVEL_COMMANDS = [
|
|
|
35
36
|
|
|
36
37
|
const SUBCOMMANDS: Record<string, string[]> = {
|
|
37
38
|
runs: ["status", "cancel", "list"],
|
|
39
|
+
tenant: ["status"],
|
|
38
40
|
harness: ["list"],
|
|
39
41
|
runtime: ["start", "status", "stop"],
|
|
40
42
|
trust: ["diff", "check"],
|
|
@@ -16,11 +16,14 @@ import { diffKxmProjectAgainstRevision, formatKxmPermissionDiff } from "../permi
|
|
|
16
16
|
import { readKxmLocalBindings, kxmUserStateRoot } from "../bindings.ts";
|
|
17
17
|
import { loadKxmProject } from "../project-config.ts";
|
|
18
18
|
import {
|
|
19
|
+
attachKxmSupervisor,
|
|
19
20
|
ensureKxmSupervisor,
|
|
20
21
|
kxmRuntimeRequest,
|
|
21
22
|
kxmSupervisorStatus,
|
|
22
23
|
} from "../runtime-supervisor.ts";
|
|
23
|
-
import { kxmRuntimePaths } from "../runtime-store.ts";
|
|
24
|
+
import { kxmRuntimePaths, runtimeError } from "../runtime-store.ts";
|
|
25
|
+
import { assembleTenantStatus, formatTenantStatus } from "../tenant-status.ts";
|
|
26
|
+
import { resolveClientAdminAuthToken } from "../hub-env.ts";
|
|
24
27
|
import {
|
|
25
28
|
formatHarnessInventory,
|
|
26
29
|
probeHarnessesAsync,
|
|
@@ -568,12 +571,14 @@ export async function cmdKxmRunList(runtime: Runtime): Promise<number> {
|
|
|
568
571
|
const projectId = String(bundle.project.value.id);
|
|
569
572
|
const supervisor = await ensureKxmSupervisor({ env: runtime.env });
|
|
570
573
|
const result = await kxmRuntimeRequest(supervisor, "GET", `/v1/projects/${encodeURIComponent(projectId)}/runs?projectRoot=${encodeURIComponent(projectRoot)}`);
|
|
571
|
-
const runs = (result.runs ?? []) as Array<{ runId: string; status: string; workflowId: string; createdAt: string }>;
|
|
574
|
+
const runs = (result.runs ?? []) as Array<{ runId: string; status: string; workflowId: string; createdAt: string; projectionError?: string }>;
|
|
572
575
|
print(
|
|
573
576
|
runtime.io,
|
|
574
577
|
runtime.json,
|
|
575
578
|
{ ok: true, command: "runs list", runs },
|
|
576
|
-
runs.length === 0
|
|
579
|
+
runs.length === 0
|
|
580
|
+
? "no runs"
|
|
581
|
+
: runs.map((run) => `${run.runId} ${run.status} ${run.workflowId} ${run.createdAt}${run.projectionError ? ` [state unverified: ${run.projectionError}]` : ""}`).join("\n"),
|
|
577
582
|
);
|
|
578
583
|
return 0;
|
|
579
584
|
} catch (error) {
|
|
@@ -586,6 +591,72 @@ export async function cmdKxmRunList(runtime: Runtime): Promise<number> {
|
|
|
586
591
|
}
|
|
587
592
|
}
|
|
588
593
|
|
|
594
|
+
export async function cmdTenantStatus(runtime: Runtime): Promise<number> {
|
|
595
|
+
let projectRoot: string | undefined;
|
|
596
|
+
let projectId: string;
|
|
597
|
+
try {
|
|
598
|
+
projectRoot = discoverKxmProjectRoot(runtime.cwd) ?? undefined;
|
|
599
|
+
if (!projectRoot) {
|
|
600
|
+
print(runtime.io, runtime.json, { ok: false, command: "tenant status", error: "project_required" }, "kxm tenant status requires a KXM project (run kxm init first)");
|
|
601
|
+
return 1;
|
|
602
|
+
}
|
|
603
|
+
const bundle = loadKxmProject(projectRoot, {});
|
|
604
|
+
projectId = String(bundle.project.value.id);
|
|
605
|
+
const root = projectRoot;
|
|
606
|
+
|
|
607
|
+
const payload = await assembleTenantStatus({
|
|
608
|
+
project: projectId,
|
|
609
|
+
hubUrl: runtime.serverUrl,
|
|
610
|
+
// Admin-scoped route: resolve the admin credential only (a project token would 401),
|
|
611
|
+
// and resolve it inside the hub source so a malformed persisted record degrades that
|
|
612
|
+
// one source instead of aborting the Runtime read with it.
|
|
613
|
+
resolveAdminToken: () => resolveClientAdminAuthToken(runtime.env),
|
|
614
|
+
fetchImpl: runtime.fetchImpl,
|
|
615
|
+
runtime: {
|
|
616
|
+
listRuns: async () => {
|
|
617
|
+
// Attach only: a status read must never conjure a supervisor. "Nothing is
|
|
618
|
+
// running" is an answer the portal can render, not a condition to repair.
|
|
619
|
+
const handle = await attachKxmSupervisor({ env: runtime.env });
|
|
620
|
+
if (!handle) {
|
|
621
|
+
throw runtimeError("runtime_supervisor_not_running", "runtime", "no live runtime supervisor on this box");
|
|
622
|
+
}
|
|
623
|
+
const result = await kxmRuntimeRequest(handle, "GET", `/v1/projects/${encodeURIComponent(projectId)}/runs?projectRoot=${encodeURIComponent(root)}`);
|
|
624
|
+
const runs = (result.runs ?? []) as Array<Record<string, unknown>>;
|
|
625
|
+
return runs.map((run) => ({
|
|
626
|
+
runId: String(run.runId ?? ""),
|
|
627
|
+
status: String(run.status ?? "unknown"),
|
|
628
|
+
homeRuntimeId: String(run.homeRuntimeId ?? ""),
|
|
629
|
+
...(typeof run.workflowId === "string" ? { workflowId: run.workflowId } : {}),
|
|
630
|
+
...(typeof run.createdAt === "string" ? { createdAt: run.createdAt } : {}),
|
|
631
|
+
...(typeof run.updatedAt === "string" ? { updatedAt: run.updatedAt } : {}),
|
|
632
|
+
...(typeof run.projectionError === "string" ? { projectionError: run.projectionError } : {}),
|
|
633
|
+
source: (typeof run.projectionError === "string" ? "runtime-cached" : "runtime-authoritative") as "runtime-cached" | "runtime-authoritative",
|
|
634
|
+
}));
|
|
635
|
+
},
|
|
636
|
+
},
|
|
637
|
+
});
|
|
638
|
+
|
|
639
|
+
if (payload.hub.state !== "ok" && payload.runtime.state !== "ok") {
|
|
640
|
+
print(
|
|
641
|
+
runtime.io,
|
|
642
|
+
runtime.json,
|
|
643
|
+
{ ok: false, command: "tenant status", error: "tenant_status_no_source", payload },
|
|
644
|
+
`neither source could be read: hub ${payload.hub.reason ?? "unknown"}, runtime ${payload.runtime.reason ?? "unknown"}`,
|
|
645
|
+
);
|
|
646
|
+
return 1;
|
|
647
|
+
}
|
|
648
|
+
print(runtime.io, runtime.json, { ok: true, command: "tenant status", ...payload }, formatTenantStatus(payload));
|
|
649
|
+
return 0;
|
|
650
|
+
} catch (error) {
|
|
651
|
+
if (error instanceof KxmConfigError) {
|
|
652
|
+
print(runtime.io, runtime.json, { ok: false, command: "tenant status", error: "tenant_status_failed", issues: error.issues }, `tenant status failed: ${error.message}`);
|
|
653
|
+
return 1;
|
|
654
|
+
}
|
|
655
|
+
print(runtime.io, runtime.json, { ok: false, command: "tenant status", error: "tenant_status_io_failed" }, "tenant status failed because a local operation did not complete");
|
|
656
|
+
return 1;
|
|
657
|
+
}
|
|
658
|
+
}
|
|
659
|
+
|
|
589
660
|
export async function cmdHarnessList(runtime: Runtime): Promise<number> {
|
|
590
661
|
const inventory = await probeHarnessesAsync({ env: runtime.env });
|
|
591
662
|
print(runtime.io, runtime.json, { ok: true, command: "harness list", ...inventory }, formatHarnessInventory(inventory));
|
package/plugins/kxm/src/cli.ts
CHANGED
|
@@ -107,6 +107,7 @@ cmdBackup,
|
|
|
107
107
|
cmdKxmRunReceipt,
|
|
108
108
|
cmdKxmRunCancel,
|
|
109
109
|
cmdKxmRunList,
|
|
110
|
+
cmdTenantStatus,
|
|
110
111
|
cmdHarnessList,
|
|
111
112
|
cmdRouteChange,
|
|
112
113
|
cmdModelsScreen,
|
|
@@ -395,6 +396,13 @@ function createProgram(ctx: CliContext, result: { code: number }): Command {
|
|
|
395
396
|
result.code = await cmdKxmRunList(runtimeFrom(ctx, this));
|
|
396
397
|
});
|
|
397
398
|
|
|
399
|
+
const tenantCmd = addGlobalOptions(program.command("tenant").description("Composed tenant reads for machine clients (portal)"));
|
|
400
|
+
tenantCmd.helpCommand("help", "Show tenant help");
|
|
401
|
+
addGlobalOptions(tenantCmd.command("status").description("Read hub metadata and authoritative Runtime run state as one labeled view"))
|
|
402
|
+
.action(async function tenantStatusAction(this: Command) {
|
|
403
|
+
result.code = await cmdTenantStatus(runtimeFrom(ctx, this));
|
|
404
|
+
});
|
|
405
|
+
|
|
398
406
|
const modelsCmd = addGlobalOptions(program.command("models").description("Manage model catalogs, roles, and route state"));
|
|
399
407
|
modelsCmd.action(async function modelsScreenAction(this: Command) { result.code = await cmdModelsScreen(runtimeFrom(ctx, this)); });
|
|
400
408
|
modelsCmd.helpCommand("help", "Show models help");
|
|
@@ -234,3 +234,16 @@ export function resolveClientHubAuthToken(env: NodeJS.ProcessEnv, project: strin
|
|
|
234
234
|
const record = readHubEnvRecord(env);
|
|
235
235
|
return record?.projectTokens?.[project]?.trim() || record?.authToken?.trim() || undefined;
|
|
236
236
|
}
|
|
237
|
+
|
|
238
|
+
/**
|
|
239
|
+
* Admin-scoped hub reads (`/v1/ops/snapshot`, …) are rejected with 401 by a project token,
|
|
240
|
+
* so this variant resolves the admin credential only: explicit `KXM_AUTH_TOKEN`, else the
|
|
241
|
+
* persisted record's admin token. It never falls back to a project token the route would
|
|
242
|
+
* refuse — handing the portal a 401 dressed as "configured" is worse than handing it none.
|
|
243
|
+
* Throws on a malformed persisted record, same as `resolveClientHubAuthToken`.
|
|
244
|
+
*/
|
|
245
|
+
export function resolveClientAdminAuthToken(env: NodeJS.ProcessEnv = process.env): string | undefined {
|
|
246
|
+
const envToken = env.KXM_AUTH_TOKEN?.trim();
|
|
247
|
+
if (envToken) return envToken;
|
|
248
|
+
return readHubEnvRecord(env)?.authToken?.trim() || undefined;
|
|
249
|
+
}
|
|
@@ -8,7 +8,7 @@ import { AGENT_COMMANDS_MAP, enforceToolPolicy, getMcpTools, reconcileInbox } fr
|
|
|
8
8
|
import { deliverInboxNotification } from "./inbox.ts";
|
|
9
9
|
import type { HubEvent, MessageRecord } from "./protocol.ts";
|
|
10
10
|
|
|
11
|
-
const VERSION = "0.7.
|
|
11
|
+
const VERSION = "0.7.57";
|
|
12
12
|
const inbox = new Map<string, MessageRecord>();
|
|
13
13
|
const notifiedInbox = new Set<string>();
|
|
14
14
|
let meshClient: HubClient | undefined;
|
|
@@ -179,6 +179,28 @@ export interface KxmSupervisorHandle {
|
|
|
179
179
|
started: boolean;
|
|
180
180
|
}
|
|
181
181
|
|
|
182
|
+
/**
|
|
183
|
+
* Attach to a live supervisor, or return `undefined`. Never starts one.
|
|
184
|
+
*
|
|
185
|
+
* `ensureKxmSupervisor` is for operators: an absent supervisor is a problem to fix, so it
|
|
186
|
+
* spawns. A poller — the portal tenant read, a status screen — has the opposite contract:
|
|
187
|
+
* polling must not conjure a daemon, and "nothing is running" is an answer to report, not
|
|
188
|
+
* a condition to repair. Returns `undefined` for every not-running shape: no claim, a
|
|
189
|
+
* supervisor registered but dead, a missing token file, or a probe that fails the
|
|
190
|
+
* token-proof challenge.
|
|
191
|
+
*/
|
|
192
|
+
export async function attachKxmSupervisor(
|
|
193
|
+
options: { stateRoot?: string; env?: NodeJS.ProcessEnv } = {},
|
|
194
|
+
): Promise<KxmSupervisorHandle | undefined> {
|
|
195
|
+
const paths = kxmRuntimePaths(options.stateRoot !== undefined ? { stateRoot: options.stateRoot } : { ...(options.env ? { env: options.env } : {}) });
|
|
196
|
+
const status = kxmSupervisorStatus(paths);
|
|
197
|
+
if (!status.running || !status.port || !status.runtimeId) return undefined;
|
|
198
|
+
const token = readKxmSupervisorToken(paths);
|
|
199
|
+
if (!token) return undefined;
|
|
200
|
+
if (!(await probeSupervisor(status.port, status.runtimeId, token))) return undefined;
|
|
201
|
+
return { runtimeId: status.runtimeId, port: status.port, token, started: false };
|
|
202
|
+
}
|
|
203
|
+
|
|
182
204
|
/** Ensure a supervisor is running: reuse a live one, otherwise auto-start. */
|
|
183
205
|
export async function ensureKxmSupervisor(
|
|
184
206
|
options: { stateRoot?: string; env?: NodeJS.ProcessEnv; spawnImpl?: (scriptPath: string, env: NodeJS.ProcessEnv) => number } = {},
|
|
@@ -658,7 +680,25 @@ async function startKxmRuntimeSupervisorInner(
|
|
|
658
680
|
sendJson(response, 400, { ok: false, error: "runtime_request_invalid", message: `project ${requestedProjectId} is not the bound project ${context.projectId}` });
|
|
659
681
|
return;
|
|
660
682
|
}
|
|
661
|
-
|
|
683
|
+
// The stored `runs` row is a cache, not the state: per-run reads fold the event
|
|
684
|
+
// log (`projectKxmRunReadOnly`) precisely because the row can be stale. A listing
|
|
685
|
+
// that returned raw rows would let every consumer — including the portal's
|
|
686
|
+
// tenant read — present cached status as authoritative. Folding replays each
|
|
687
|
+
// run's events; workflows are transition-bounded, so this stays cheap at the
|
|
688
|
+
// 50-run cap. A run that refuses to fold is returned with its cached row plus
|
|
689
|
+
// `projectionError`, so one corrupt run cannot make the listing lie by omission.
|
|
690
|
+
const runs = context.eventStore.runsForProject(requestedProjectId, 50).map((stored) => {
|
|
691
|
+
try {
|
|
692
|
+
return projectKxmRunReadOnly(context, stored.runId);
|
|
693
|
+
} catch (error) {
|
|
694
|
+
return {
|
|
695
|
+
...stored,
|
|
696
|
+
projectionError: error instanceof KxmConfigError
|
|
697
|
+
? (error.issues[0]?.code ?? "runtime_projection_failed")
|
|
698
|
+
: "runtime_projection_failed",
|
|
699
|
+
};
|
|
700
|
+
}
|
|
701
|
+
});
|
|
662
702
|
sendJson(response, 200, { ok: true, runs });
|
|
663
703
|
return;
|
|
664
704
|
}
|
|
@@ -0,0 +1,371 @@
|
|
|
1
|
+
import { hubBindingScope } from "./hub-binding.ts";
|
|
2
|
+
|
|
3
|
+
export const TENANT_STATUS_SCHEMA = "kxm.tenant-status.v1" as const;
|
|
4
|
+
|
|
5
|
+
/**
|
|
6
|
+
* One upstream of the tenant view, with its own reachability. A source that could not be
|
|
7
|
+
* read is reported as `unavailable` with a stable reason — never silently filled from the
|
|
8
|
+
* other source, because the two sources answer different questions: the hub holds
|
|
9
|
+
* *metadata* (agent roster, message queue, its own workflow runs), the Runtime holds the
|
|
10
|
+
* *authoritative* run state on the box that owns the run. Blurring them is how a portal
|
|
11
|
+
* ends up rendering one as the other.
|
|
12
|
+
*/
|
|
13
|
+
export type TenantSourceState = "ok" | "unavailable";
|
|
14
|
+
|
|
15
|
+
export interface TenantSource<T> {
|
|
16
|
+
state: TenantSourceState;
|
|
17
|
+
/** When this source was read (or attempted), so a consumer can age the data itself. */
|
|
18
|
+
observedAt: string;
|
|
19
|
+
/** Stable machine code (`hub_unreachable`, `hub_timeout`, `runtime_supervisor_not_running`, …). */
|
|
20
|
+
reason?: string;
|
|
21
|
+
value?: T;
|
|
22
|
+
}
|
|
23
|
+
|
|
24
|
+
/** Hub-side roster entry: presence and liveness, per the hub's own observation. */
|
|
25
|
+
export interface TenantHubAgent {
|
|
26
|
+
id: string;
|
|
27
|
+
name: string;
|
|
28
|
+
online: boolean;
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
export interface TenantHubStage {
|
|
32
|
+
id: string;
|
|
33
|
+
status: string;
|
|
34
|
+
attempts?: number | undefined;
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
/**
|
|
38
|
+
* A run as the hub records it. The hub mints its own run ids at workflow start; this is a
|
|
39
|
+
* hub workflow run — metadata about work, not the Runtime's state for it.
|
|
40
|
+
*/
|
|
41
|
+
export interface TenantHubRun {
|
|
42
|
+
id: string;
|
|
43
|
+
status: string;
|
|
44
|
+
definitionId: string;
|
|
45
|
+
currentStage?: string | undefined;
|
|
46
|
+
targetAgentName?: string | undefined;
|
|
47
|
+
updatedAt?: string | undefined;
|
|
48
|
+
progress?: { done: number; total: number };
|
|
49
|
+
stages?: TenantHubStage[];
|
|
50
|
+
source: "hub-projection";
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
export interface TenantHubPlan {
|
|
54
|
+
id: string;
|
|
55
|
+
runId: string;
|
|
56
|
+
summary: string;
|
|
57
|
+
createdAt: string;
|
|
58
|
+
stageId?: string | undefined;
|
|
59
|
+
severity?: string | undefined;
|
|
60
|
+
}
|
|
61
|
+
|
|
62
|
+
export interface TenantHubValue {
|
|
63
|
+
project: string;
|
|
64
|
+
fetchedAt: string;
|
|
65
|
+
agents: TenantHubAgent[];
|
|
66
|
+
openMessageTotal: number;
|
|
67
|
+
runTotal: number;
|
|
68
|
+
runs: TenantHubRun[];
|
|
69
|
+
plans: TenantHubPlan[];
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
/** A run as the Runtime that owns it records it. This is the authoritative state. */
|
|
73
|
+
export interface TenantRuntimeRun {
|
|
74
|
+
runId: string;
|
|
75
|
+
status: string;
|
|
76
|
+
homeRuntimeId: string;
|
|
77
|
+
workflowId?: string | undefined;
|
|
78
|
+
createdAt?: string | undefined;
|
|
79
|
+
updatedAt?: string | undefined;
|
|
80
|
+
/** Set when the stored row exists but the event-log fold refused; `status` is then the
|
|
81
|
+
* cached row, and a consumer must not present it as folded state. */
|
|
82
|
+
projectionError?: string | undefined;
|
|
83
|
+
/** `runtime-cached` exactly when `projectionError` is set: the row is the cache, not the
|
|
84
|
+
* folded state, and neither the label nor the cross-check may treat it as authoritative. */
|
|
85
|
+
source: "runtime-authoritative" | "runtime-cached";
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
export interface TenantRuntimeValue {
|
|
89
|
+
runs: TenantRuntimeRun[];
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
export interface TenantRunDiscrepancy {
|
|
93
|
+
runId: string;
|
|
94
|
+
hubStatus: string;
|
|
95
|
+
runtimeStatus: string;
|
|
96
|
+
}
|
|
97
|
+
|
|
98
|
+
/**
|
|
99
|
+
* Cross-check between the two run populations. Hub runs and Runtime runs mint their ids
|
|
100
|
+
* independently (`newId("run")` on each side), so an id appearing in both means a hub run
|
|
101
|
+
* that this box drove — and only then is a status comparison meaningful. With no overlap
|
|
102
|
+
* the comparison is **unverified, not agreed**: "no matching ids" says the populations did
|
|
103
|
+
* not intersect, nothing more, and reporting agreement there would fabricate a green light
|
|
104
|
+
* out of an empty set.
|
|
105
|
+
*/
|
|
106
|
+
export interface TenantRunComparison {
|
|
107
|
+
state: "compared" | "unverified" | "unavailable";
|
|
108
|
+
reason?: string;
|
|
109
|
+
matched?: number;
|
|
110
|
+
/** Shared ids whose Runtime row failed to fold: present in both stores, but the
|
|
111
|
+
* authoritative side could not be read, so those runs are unverified rather than agreed. */
|
|
112
|
+
unverifiedFoldRuns?: number;
|
|
113
|
+
discrepancies?: TenantRunDiscrepancy[];
|
|
114
|
+
}
|
|
115
|
+
|
|
116
|
+
export interface TenantStatusPayload {
|
|
117
|
+
schema: typeof TENANT_STATUS_SCHEMA;
|
|
118
|
+
project: string;
|
|
119
|
+
generatedAt: string;
|
|
120
|
+
hubUrl: string;
|
|
121
|
+
/** Reuses the `kxm hub bind` classification: `loopback` and `remote` ship different rules. */
|
|
122
|
+
bindingScope: "loopback" | "remote";
|
|
123
|
+
hub: TenantSource<TenantHubValue>;
|
|
124
|
+
runtime: TenantSource<TenantRuntimeValue>;
|
|
125
|
+
runComparison: TenantRunComparison;
|
|
126
|
+
/** True when any source is unavailable. Assembling a payload is not the same as seeing. */
|
|
127
|
+
degraded: boolean;
|
|
128
|
+
}
|
|
129
|
+
|
|
130
|
+
export interface TenantStatusRuntimeReader {
|
|
131
|
+
listRuns(): Promise<TenantRuntimeRun[]>;
|
|
132
|
+
}
|
|
133
|
+
|
|
134
|
+
const DEFAULT_HUB_TIMEOUT_MS = 5_000;
|
|
135
|
+
|
|
136
|
+
function hubSourceReason(status: number): string {
|
|
137
|
+
if (status === 401 || status === 403) return "hub_unauthorized";
|
|
138
|
+
if (status === 404) return "hub_not_found";
|
|
139
|
+
return `hub_http_${status}`;
|
|
140
|
+
}
|
|
141
|
+
|
|
142
|
+
function asString(value: unknown): string | undefined {
|
|
143
|
+
return typeof value === "string" && value.length > 0 ? value : undefined;
|
|
144
|
+
}
|
|
145
|
+
|
|
146
|
+
function isObject(value: unknown): value is Record<string, unknown> {
|
|
147
|
+
return typeof value === "object" && value !== null && !Array.isArray(value);
|
|
148
|
+
}
|
|
149
|
+
|
|
150
|
+
/**
|
|
151
|
+
* Compose the tenant view from the two authorities that actually hold the data.
|
|
152
|
+
*
|
|
153
|
+
* Both sources are read **concurrently and independently**: a hub that hangs to its
|
|
154
|
+
* deadline cannot delay or suppress the Runtime half of the view, and a Runtime that is
|
|
155
|
+
* not running cannot take the hub metadata down with it. The hub is read over HTTP
|
|
156
|
+
* (`/v1/ops/snapshot`, admin credential); the Runtime is read through an injected reader
|
|
157
|
+
* so this function stays pure and testable — the CLI passes a reader that attaches to the
|
|
158
|
+
* live supervisor, and a test passes whatever it likes.
|
|
159
|
+
*/
|
|
160
|
+
export async function assembleTenantStatus(input: {
|
|
161
|
+
project: string;
|
|
162
|
+
hubUrl: string;
|
|
163
|
+
/** Resolves the admin credential. May throw on a malformed persisted record; that
|
|
164
|
+
* failure belongs to the hub source alone and never aborts the Runtime read. */
|
|
165
|
+
resolveAdminToken: () => string | undefined;
|
|
166
|
+
fetchImpl: typeof fetch;
|
|
167
|
+
runtime: TenantStatusRuntimeReader;
|
|
168
|
+
hubTimeoutMs?: number | undefined;
|
|
169
|
+
now?: () => Date;
|
|
170
|
+
}): Promise<TenantStatusPayload> {
|
|
171
|
+
const now = input.now ?? ((): Date => new Date());
|
|
172
|
+
const base = input.hubUrl.replace(/\/$/, "");
|
|
173
|
+
const at = (): string => now().toISOString();
|
|
174
|
+
|
|
175
|
+
const readHub = async (): Promise<TenantSource<TenantHubValue>> => {
|
|
176
|
+
const attemptedAt = at();
|
|
177
|
+
let token: string | undefined;
|
|
178
|
+
try {
|
|
179
|
+
token = input.resolveAdminToken();
|
|
180
|
+
} catch (error) {
|
|
181
|
+
// A malformed hub-env record is a configuration failure with a specific repair.
|
|
182
|
+
// It belongs to this source; the Runtime read proceeds regardless.
|
|
183
|
+
void error;
|
|
184
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: "hub_credential_unreadable" };
|
|
185
|
+
}
|
|
186
|
+
try {
|
|
187
|
+
const response = await input.fetchImpl(`${base}/v1/ops/snapshot?project=${encodeURIComponent(input.project)}`, {
|
|
188
|
+
headers: token ? { authorization: `Bearer ${token}` } : {},
|
|
189
|
+
signal: AbortSignal.timeout(input.hubTimeoutMs ?? DEFAULT_HUB_TIMEOUT_MS),
|
|
190
|
+
});
|
|
191
|
+
if (!response.ok) {
|
|
192
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: hubSourceReason(response.status) };
|
|
193
|
+
}
|
|
194
|
+
let body: unknown;
|
|
195
|
+
try {
|
|
196
|
+
body = await response.json();
|
|
197
|
+
} catch (error) {
|
|
198
|
+
// A deadline can expire mid-body; that is a timeout, not malformed content, and
|
|
199
|
+
// classifying it as `hub_response_invalid` would hide the one fact that matters.
|
|
200
|
+
const name = error instanceof Error ? error.name : undefined;
|
|
201
|
+
if (name === "TimeoutError" || name === "AbortError") {
|
|
202
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: "hub_timeout" };
|
|
203
|
+
}
|
|
204
|
+
// Reachable but not a JSON document: a different failure than a dead socket.
|
|
205
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: "hub_response_invalid" };
|
|
206
|
+
}
|
|
207
|
+
if (!isObject(body)
|
|
208
|
+
|| typeof body.project !== "string"
|
|
209
|
+
|| typeof body.fetchedAt !== "string"
|
|
210
|
+
|| !Array.isArray(body.agents)
|
|
211
|
+
|| !Array.isArray(body.runs)
|
|
212
|
+
|| !Array.isArray(body.plans ?? [])
|
|
213
|
+
|| typeof body.openMessageTotal !== "number"
|
|
214
|
+
|| typeof body.runTotal !== "number") {
|
|
215
|
+
// A 200 with an unusable body must not become a healthy empty snapshot.
|
|
216
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: "hub_response_invalid" };
|
|
217
|
+
}
|
|
218
|
+
if (body.project !== input.project) {
|
|
219
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: "hub_project_mismatch" };
|
|
220
|
+
}
|
|
221
|
+
const runRows = body.runs.filter(isObject);
|
|
222
|
+
const value: TenantHubValue = {
|
|
223
|
+
project: body.project,
|
|
224
|
+
fetchedAt: body.fetchedAt,
|
|
225
|
+
agents: body.agents.filter(isObject).map((agent) => ({
|
|
226
|
+
id: asString(agent.id) ?? "",
|
|
227
|
+
name: asString(agent.name) ?? "",
|
|
228
|
+
online: agent.online === true,
|
|
229
|
+
})),
|
|
230
|
+
openMessageTotal: body.openMessageTotal,
|
|
231
|
+
runTotal: body.runTotal,
|
|
232
|
+
runs: runRows.map((run) => {
|
|
233
|
+
const stages = Array.isArray(run.stages)
|
|
234
|
+
? run.stages.filter(isObject).map((stage) => ({
|
|
235
|
+
id: asString(stage.id) ?? "",
|
|
236
|
+
status: asString(stage.status) ?? "unknown",
|
|
237
|
+
...(typeof stage.attempts === "number" ? { attempts: stage.attempts } : {}),
|
|
238
|
+
}))
|
|
239
|
+
: undefined;
|
|
240
|
+
const done = stages?.filter((stage) => stage.status === "passed" || stage.status === "failed" || stage.status === "warning").length;
|
|
241
|
+
return {
|
|
242
|
+
id: asString(run.id) ?? "",
|
|
243
|
+
status: asString(run.status) ?? "unknown",
|
|
244
|
+
definitionId: asString(run.definitionId) ?? "",
|
|
245
|
+
...(asString(run.currentStage) ? { currentStage: asString(run.currentStage) } : {}),
|
|
246
|
+
...(asString(run.targetAgentName) ? { targetAgentName: asString(run.targetAgentName) } : {}),
|
|
247
|
+
...(asString(run.updatedAt) ? { updatedAt: asString(run.updatedAt) } : {}),
|
|
248
|
+
...(stages !== undefined ? { progress: { done: done ?? 0, total: stages.length }, stages } : {}),
|
|
249
|
+
source: "hub-projection" as const,
|
|
250
|
+
};
|
|
251
|
+
}),
|
|
252
|
+
plans: (Array.isArray(body.plans) ? body.plans : []).filter(isObject).map((plan) => ({
|
|
253
|
+
id: asString(plan.id) ?? "",
|
|
254
|
+
runId: asString(plan.runId) ?? "",
|
|
255
|
+
summary: asString(plan.summary) ?? "",
|
|
256
|
+
createdAt: asString(plan.createdAt) ?? "",
|
|
257
|
+
...(asString(plan.stageId) ? { stageId: asString(plan.stageId) } : {}),
|
|
258
|
+
...(asString(plan.severity) ? { severity: asString(plan.severity) } : {}),
|
|
259
|
+
})),
|
|
260
|
+
};
|
|
261
|
+
return { state: "ok", observedAt: attemptedAt, value };
|
|
262
|
+
} catch (error) {
|
|
263
|
+
const name = error instanceof Error ? error.name : undefined;
|
|
264
|
+
if (name === "TimeoutError" || name === "AbortError") {
|
|
265
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: "hub_timeout" };
|
|
266
|
+
}
|
|
267
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: "hub_unreachable" };
|
|
268
|
+
}
|
|
269
|
+
};
|
|
270
|
+
|
|
271
|
+
const readRuntime = async (): Promise<TenantSource<TenantRuntimeValue>> => {
|
|
272
|
+
const attemptedAt = at();
|
|
273
|
+
try {
|
|
274
|
+
const runs = await input.runtime.listRuns();
|
|
275
|
+
return { state: "ok", observedAt: attemptedAt, value: { runs } };
|
|
276
|
+
} catch (error) {
|
|
277
|
+
const code = error instanceof Error && "issues" in error
|
|
278
|
+
&& Array.isArray((error as { issues: Array<{ code?: unknown }> }).issues)
|
|
279
|
+
&& typeof (error as { issues: Array<{ code?: unknown }> }).issues[0]?.code === "string"
|
|
280
|
+
? (error as { issues: Array<{ code: string }> }).issues[0]!.code
|
|
281
|
+
: undefined;
|
|
282
|
+
return { state: "unavailable", observedAt: attemptedAt, reason: code ?? "runtime_unavailable" };
|
|
283
|
+
}
|
|
284
|
+
};
|
|
285
|
+
|
|
286
|
+
const [hub, runtime] = await Promise.all([readHub(), readRuntime()]);
|
|
287
|
+
|
|
288
|
+
let runComparison: TenantRunComparison;
|
|
289
|
+
if (!hub.value || !runtime.value) {
|
|
290
|
+
runComparison = {
|
|
291
|
+
state: "unavailable",
|
|
292
|
+
reason: !hub.value ? `hub_${hub.reason ?? "unavailable"}` : `runtime_${runtime.reason ?? "unavailable"}`,
|
|
293
|
+
};
|
|
294
|
+
} else {
|
|
295
|
+
// Only cleanly folded rows are authoritative. A row whose fold failed is the cache, and
|
|
296
|
+
// counting it as agreement is precisely the lie this comparison exists to prevent: a
|
|
297
|
+
// corrupt event log under a hub run's id would otherwise print "agree".
|
|
298
|
+
const authoritative = new Map(
|
|
299
|
+
runtime.value.runs
|
|
300
|
+
.filter((run) => run.projectionError === undefined)
|
|
301
|
+
.map((run) => [run.runId, run.status] as const),
|
|
302
|
+
);
|
|
303
|
+
const hubIds = new Set(hub.value.runs.map((run) => run.id));
|
|
304
|
+
const foldFailed = runtime.value.runs.filter((run) => run.projectionError !== undefined && hubIds.has(run.runId));
|
|
305
|
+
const discrepancies: TenantRunDiscrepancy[] = [];
|
|
306
|
+
let matched = 0;
|
|
307
|
+
for (const projected of hub.value.runs) {
|
|
308
|
+
const runtimeStatus = authoritative.get(projected.id);
|
|
309
|
+
if (runtimeStatus === undefined) continue;
|
|
310
|
+
matched += 1;
|
|
311
|
+
if (runtimeStatus !== projected.status) {
|
|
312
|
+
discrepancies.push({ runId: projected.id, hubStatus: projected.status, runtimeStatus });
|
|
313
|
+
}
|
|
314
|
+
}
|
|
315
|
+
if (matched > 0) {
|
|
316
|
+
runComparison = {
|
|
317
|
+
state: "compared",
|
|
318
|
+
matched,
|
|
319
|
+
...(foldFailed.length > 0 ? { unverifiedFoldRuns: foldFailed.length } : {}),
|
|
320
|
+
...(discrepancies.length > 0 ? { discrepancies } : {}),
|
|
321
|
+
};
|
|
322
|
+
} else {
|
|
323
|
+
runComparison = foldFailed.length > 0
|
|
324
|
+
? { state: "unverified", reason: "runtime_fold_failed", matched: 0, unverifiedFoldRuns: foldFailed.length }
|
|
325
|
+
: { state: "unverified", reason: "run_identity_link_absent", matched: 0 };
|
|
326
|
+
}
|
|
327
|
+
}
|
|
328
|
+
|
|
329
|
+
return {
|
|
330
|
+
schema: TENANT_STATUS_SCHEMA,
|
|
331
|
+
project: input.project,
|
|
332
|
+
generatedAt: at(),
|
|
333
|
+
hubUrl: input.hubUrl,
|
|
334
|
+
bindingScope: hubBindingScope(input.hubUrl),
|
|
335
|
+
hub,
|
|
336
|
+
runtime,
|
|
337
|
+
runComparison,
|
|
338
|
+
degraded: hub.state !== "ok" || runtime.state !== "ok",
|
|
339
|
+
};
|
|
340
|
+
}
|
|
341
|
+
|
|
342
|
+
/** One line per source for humans; the portal reads the JSON, operators read this. */
|
|
343
|
+
export function formatTenantStatus(payload: TenantStatusPayload): string {
|
|
344
|
+
const hubLine = payload.hub.state === "ok" && payload.hub.value
|
|
345
|
+
? `hub ${payload.hub.value.agents.filter((agent) => agent.online).length}/${payload.hub.value.agents.length} agents online, ${payload.hub.value.runs.length} hub runs, ${payload.hub.value.openMessageTotal} open messages`
|
|
346
|
+
: `hub unavailable (${payload.hub.reason ?? "unknown"})`;
|
|
347
|
+
const runtimeLine = payload.runtime.state === "ok" && payload.runtime.value
|
|
348
|
+
? (() => {
|
|
349
|
+
const cached = payload.runtime.value!.runs.filter((run) => run.projectionError !== undefined).length;
|
|
350
|
+
const authoritative = payload.runtime.value!.runs.length - cached;
|
|
351
|
+
// A fold-failed row is the cache; printing it under "authoritative" would put the
|
|
352
|
+
// lie back in exactly the place an operator reads first.
|
|
353
|
+
return `runtime ${authoritative} runs (authoritative, on this box)${cached > 0 ? `, ${cached} cached (fold failed, not state)` : ""}`;
|
|
354
|
+
})()
|
|
355
|
+
: `runtime unavailable (${payload.runtime.reason ?? "unknown"})`;
|
|
356
|
+
const comparisonLine = payload.runComparison.state === "compared"
|
|
357
|
+
? payload.runComparison.discrepancies && payload.runComparison.discrepancies.length > 0
|
|
358
|
+
? `cross-check: ${payload.runComparison.discrepancies.length} of ${payload.runComparison.matched} matched run(s) disagree${payload.runComparison.unverifiedFoldRuns ? ` (${payload.runComparison.unverifiedFoldRuns} unverified: fold failed)` : ""}`
|
|
359
|
+
: `cross-check: ${payload.runComparison.matched} matched run(s) agree${payload.runComparison.unverifiedFoldRuns ? ` (${payload.runComparison.unverifiedFoldRuns} unverified: fold failed)` : ""}`
|
|
360
|
+
: payload.runComparison.state === "unverified"
|
|
361
|
+
? payload.runComparison.reason === "runtime_fold_failed"
|
|
362
|
+
? `cross-check: unavailable — ${payload.runComparison.unverifiedFoldRuns ?? 0} shared run(s) could not be verified (runtime fold failed)`
|
|
363
|
+
: "cross-check: unavailable — no run id appears in both sources (independent id spaces)"
|
|
364
|
+
: `cross-check: unavailable (${payload.runComparison.reason ?? "unknown"})`;
|
|
365
|
+
return [
|
|
366
|
+
`tenant ${payload.project} @ ${payload.hubUrl} (${payload.bindingScope})`,
|
|
367
|
+
hubLine,
|
|
368
|
+
runtimeLine,
|
|
369
|
+
comparisonLine,
|
|
370
|
+
].join("\n");
|
|
371
|
+
}
|