@takk/bayesoutputgate 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +92 -0
- package/LICENSE +190 -0
- package/NOTICE +45 -0
- package/README.md +403 -0
- package/SECURITY.md +98 -0
- package/SPEC.md +467 -0
- package/dist/adapter/index.cjs +411 -0
- package/dist/adapter/index.d.cts +29 -0
- package/dist/adapter/index.d.ts +29 -0
- package/dist/adapter/index.js +404 -0
- package/dist/audit/index.cjs +82 -0
- package/dist/audit/index.d.cts +40 -0
- package/dist/audit/index.d.ts +40 -0
- package/dist/audit/index.js +77 -0
- package/dist/bayesfactor/index.cjs +152 -0
- package/dist/bayesfactor/index.d.cts +15 -0
- package/dist/bayesfactor/index.d.ts +15 -0
- package/dist/bayesfactor/index.js +149 -0
- package/dist/beta/index.cjs +180 -0
- package/dist/beta/index.d.cts +45 -0
- package/dist/beta/index.d.ts +45 -0
- package/dist/beta/index.js +178 -0
- package/dist/calibration/index.cjs +339 -0
- package/dist/calibration/index.d.cts +53 -0
- package/dist/calibration/index.d.ts +53 -0
- package/dist/calibration/index.js +333 -0
- package/dist/cli/index.cjs +968 -0
- package/dist/cli/index.d.cts +1 -0
- package/dist/cli/index.d.ts +1 -0
- package/dist/cli/index.js +966 -0
- package/dist/dimensions/index.cjs +106 -0
- package/dist/dimensions/index.d.cts +33 -0
- package/dist/dimensions/index.d.ts +33 -0
- package/dist/dimensions/index.js +104 -0
- package/dist/edge/index.cjs +1141 -0
- package/dist/edge/index.d.cts +12 -0
- package/dist/edge/index.d.ts +12 -0
- package/dist/edge/index.js +1109 -0
- package/dist/gate/index.cjs +803 -0
- package/dist/gate/index.d.cts +77 -0
- package/dist/gate/index.d.ts +77 -0
- package/dist/gate/index.js +799 -0
- package/dist/hypothesis/index.cjs +268 -0
- package/dist/hypothesis/index.d.cts +38 -0
- package/dist/hypothesis/index.d.ts +38 -0
- package/dist/hypothesis/index.js +266 -0
- package/dist/index.cjs +1141 -0
- package/dist/index.d.cts +29 -0
- package/dist/index.d.ts +29 -0
- package/dist/index.js +1109 -0
- package/dist/likelihood/index.cjs +137 -0
- package/dist/likelihood/index.d.cts +23 -0
- package/dist/likelihood/index.d.ts +23 -0
- package/dist/likelihood/index.js +132 -0
- package/dist/node/index.cjs +1282 -0
- package/dist/node/index.d.cts +24 -0
- package/dist/node/index.d.ts +24 -0
- package/dist/node/index.js +1246 -0
- package/dist/policy/index.cjs +88 -0
- package/dist/policy/index.d.cts +11 -0
- package/dist/policy/index.d.ts +11 -0
- package/dist/policy/index.js +85 -0
- package/dist/types-bMjn1j4e.d.cts +159 -0
- package/dist/types-bMjn1j4e.d.ts +159 -0
- package/package.json +142 -0
package/SPEC.md
ADDED
|
@@ -0,0 +1,467 @@
|
|
|
1
|
+
# @takk/bayesoutputgate - Technical Specification
|
|
2
|
+
|
|
3
|
+
**Version:** 1.0.0
|
|
4
|
+
**Status:** Stable
|
|
5
|
+
**License:** Apache-2.0
|
|
6
|
+
|
|
7
|
+
This document is the binding contract between `@takk/bayesoutputgate` and its consumers. Behavior described here is covered by SemVer: breaking changes require a major version bump and a deprecation cycle (see [SEMVER POLICY](#52-semver-policy)).
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## 1. Purpose
|
|
12
|
+
|
|
13
|
+
BayesOutputGate is a universal, zero-runtime-dependency library and CLI that turns the quality scores of an output into a calibrated pass, fail, or escalate decision. You feed in the per-dimension scores of an output, from any scorer (an LLM-as-judge, a classifier, a regex), and the engine answers the question a reviewer actually asks, transparently:
|
|
14
|
+
|
|
15
|
+
- **Calibrating** two hypotheses from labeled history: a Beta model of the scores of known-high-quality outputs and a Beta model of the scores of known-low-quality outputs, per dimension, regularized by a prior and updatable online.
|
|
16
|
+
- **Weighing** a new output's scores with a Bayes Factor: the ratio of the likelihood under the high-quality hypothesis to the likelihood under the low-quality one, summed in log space across weighted dimensions.
|
|
17
|
+
- **Interpreting** the combined Bayes Factor on the Jeffreys scale, whose symmetric boundaries at 3, 10, and 100 separate inconclusive, substantial, strong, and decisive evidence.
|
|
18
|
+
- **Deciding** pass, fail, or escalate, by a pure Bayes Factor policy on that scale, or by a decision-theoretic policy that minimizes expected loss under an asymmetric cost of the two error types.
|
|
19
|
+
- **Diagnosing** trust: a goodness-of-fit test of the Beta assumption, a dependence diagnostic over the dimensions, and measured decision calibration (Brier score, expected calibration error, reliability diagram).
|
|
20
|
+
- **Proving** the decision trail with a tamper-evident hash-chained audit log.
|
|
21
|
+
|
|
22
|
+
It is library-shaped, not service-shaped. There is no central server, no SaaS dependency, no SDK lock-in. The core is node-free; it runs in Node, edge runtimes, and the browser. It is an output-validation engine for Massive Intelligence (IM) systems and for any setting where a stream of quality scores must be turned into calibrated accept or reject decisions.
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## 2. Public surface
|
|
27
|
+
|
|
28
|
+
### 2.1 Entry points
|
|
29
|
+
|
|
30
|
+
The package ships thirteen functional subpath exports plus `./package.json`, each with separate `import` (ESM) and `require` (CJS) conditions and matching `.d.ts` / `.d.cts` files:
|
|
31
|
+
|
|
32
|
+
| Subpath | Use |
|
|
33
|
+
|---|---|
|
|
34
|
+
| `.` | The `evaluate` gate, `OutputGate`, `OutputGateMonitor`, plus the full toolkit and types |
|
|
35
|
+
| `./beta` | `BetaModel`, the calibrated, online-updatable Beta model both hypotheses share |
|
|
36
|
+
| `./hypothesis` | `HypothesisManager`, per-dimension high and low score models from labeled history |
|
|
37
|
+
| `./likelihood` | `logScoreLikelihood`, `logBinaryMarginalLikelihood`, `logVectorLikelihood`, `toScoreMap` |
|
|
38
|
+
| `./bayesfactor` | `bayesFactor`, `jeffreysStrength`, the combined Bayes Factor and its interpretation |
|
|
39
|
+
| `./dimensions` | `dependenceDiagnostic`, pairwise correlation between dimensions |
|
|
40
|
+
| `./policy` | `decide`, `posteriorHighQuality`, the Bayes Factor and decision-theoretic policies |
|
|
41
|
+
| `./calibration` | `goodnessOfFit`, `brierScore`, `expectedCalibrationError`, `reliability` |
|
|
42
|
+
| `./gate` | `evaluate`, `OutputGate`, `OutputGateMonitor` |
|
|
43
|
+
| `./audit` | `AuditChain`, `verifyChain`, `sha256Hex`, an append-only SHA-256 hash chain over Web Crypto |
|
|
44
|
+
| `./adapter` | `bayesOutputGateTool`, `runTool`, `describeTool`, the framework-agnostic MCP and LLM tool |
|
|
45
|
+
| `./node` | File loaders for JSON and CSV score history (the only Node-only entry) |
|
|
46
|
+
| `./edge` | The node-free core, re-exported for edge runtimes and the browser |
|
|
47
|
+
| `./package.json` | Manifest access for tooling |
|
|
48
|
+
|
|
49
|
+
A `bayesoutputgate` binary is exposed via `package.json#bin -> ./dist/cli/index.js`.
|
|
50
|
+
|
|
51
|
+
### 2.2 Core types
|
|
52
|
+
|
|
53
|
+
```ts
|
|
54
|
+
type HypothesisKind = "high" | "low";
|
|
55
|
+
type GateAction = "pass" | "fail" | "escalate";
|
|
56
|
+
|
|
57
|
+
interface QualityScore {
|
|
58
|
+
readonly dimension: string; // the axis this score measures, for example "factuality"
|
|
59
|
+
readonly value: number; // the score in [0, 1], higher is better
|
|
60
|
+
}
|
|
61
|
+
type ScoreVector = readonly QualityScore[];
|
|
62
|
+
|
|
63
|
+
interface BetaParams {
|
|
64
|
+
readonly a: number; // positive
|
|
65
|
+
readonly b: number; // positive
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
interface DimensionModel {
|
|
69
|
+
readonly dimension: string;
|
|
70
|
+
readonly high: BetaParams; // Beta over the scores of known-high-quality outputs
|
|
71
|
+
readonly low: BetaParams; // Beta over the scores of known-low-quality outputs
|
|
72
|
+
readonly weight: number; // non-negative weight on this dimension's log-Bayes-Factor
|
|
73
|
+
}
|
|
74
|
+
|
|
75
|
+
interface LabeledObservation {
|
|
76
|
+
readonly scores: ScoreVector;
|
|
77
|
+
readonly label: HypothesisKind; // the known ground-truth quality of this output
|
|
78
|
+
}
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
### 2.3 Core API
|
|
82
|
+
|
|
83
|
+
#### `evaluate(scores, models, policy, guards?): GateDecision`
|
|
84
|
+
|
|
85
|
+
Weighs one output's scores against the dimension models under a policy, in a single call: it computes the Bayes Factor, applies the policy, and, when guards with a precomputed assumption report are supplied, escalates instead of trusting a Bayes Factor from a misspecified model.
|
|
86
|
+
|
|
87
|
+
```ts
|
|
88
|
+
function evaluate(
|
|
89
|
+
scores: ScoreVector,
|
|
90
|
+
models: readonly DimensionModel[],
|
|
91
|
+
policy: PolicyConfig,
|
|
92
|
+
guards?: GateGuards,
|
|
93
|
+
): GateDecision;
|
|
94
|
+
|
|
95
|
+
interface GateDecision {
|
|
96
|
+
readonly action: GateAction;
|
|
97
|
+
readonly bayesFactor: number;
|
|
98
|
+
readonly logBayesFactor: number;
|
|
99
|
+
readonly strength: EvidenceStrength;
|
|
100
|
+
readonly rationale: "bayes-factor" | "expected-loss" | "assumption-violated";
|
|
101
|
+
readonly posteriorHighQuality?: number; // present under a decision-theoretic policy
|
|
102
|
+
readonly expectedLoss?: ExpectedLoss; // present under a decision-theoretic policy
|
|
103
|
+
readonly assumptions?: AssumptionReport; // present when guards supplied a report
|
|
104
|
+
readonly contributions: readonly DimensionContribution[];
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
interface GateGuards {
|
|
108
|
+
assumptions?: AssumptionReport; // a precomputed report; see assessAssumptions
|
|
109
|
+
requireGoodnessOfFit?: boolean; // escalate if goodness-of-fit is inadequate
|
|
110
|
+
requireIndependence?: boolean; // escalate if the independence assumption is unsafe
|
|
111
|
+
}
|
|
112
|
+
|
|
113
|
+
interface AssumptionReport {
|
|
114
|
+
readonly goodnessOfFitAdequate: boolean;
|
|
115
|
+
readonly independenceAssumptionSafe: boolean;
|
|
116
|
+
readonly inadequateDimensions: readonly string[];
|
|
117
|
+
readonly dependentPairs: readonly string[]; // each "a~b"
|
|
118
|
+
}
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
When a `require` flag is set and the matching assumption fails, the decision's action becomes `escalate` and its rationale `"assumption-violated"`, while the Bayes Factor, strength, and contributions are preserved for the reviewer. `OutputGate` and `OutputGateMonitor` accept guards in their options; the monitor reassesses the report from the history on every batch `fit`.
|
|
122
|
+
|
|
123
|
+
#### `bayesFactor(scores, models): BayesFactorResult`
|
|
124
|
+
|
|
125
|
+
Computes the combined natural-log Bayes Factor in favor of high quality as the weighted sum of per-dimension Beta log-density differences, over the dimensions that have both a score and a model, and interprets it on the Jeffreys scale.
|
|
126
|
+
|
|
127
|
+
```ts
|
|
128
|
+
type EvidenceStrength =
|
|
129
|
+
| "decisive-low" | "strong-low" | "substantial-low"
|
|
130
|
+
| "inconclusive"
|
|
131
|
+
| "substantial-high" | "strong-high" | "decisive-high";
|
|
132
|
+
|
|
133
|
+
interface DimensionContribution {
|
|
134
|
+
readonly dimension: string;
|
|
135
|
+
readonly score: number;
|
|
136
|
+
readonly logBayesFactor: number;
|
|
137
|
+
readonly weight: number;
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
interface BayesFactorResult {
|
|
141
|
+
readonly logBayesFactor: number;
|
|
142
|
+
readonly bayesFactor: number; // exp(logBayesFactor), may be Infinity
|
|
143
|
+
readonly favored: "high" | "low" | "inconclusive";
|
|
144
|
+
readonly strength: EvidenceStrength;
|
|
145
|
+
readonly contributions: readonly DimensionContribution[];
|
|
146
|
+
}
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
#### `HypothesisManager` (`./hypothesis`)
|
|
150
|
+
|
|
151
|
+
Keeps two calibrated `BetaModel`s per dimension, one for known-high and one for known-low outputs, fit from labeled history and updatable online, and produces the `DimensionModel[]` the Bayes Factor consumes.
|
|
152
|
+
|
|
153
|
+
```ts
|
|
154
|
+
new HypothesisManager(options?: {
|
|
155
|
+
dimensions?: ReadonlyArray<{ dimension: string; weight?: number; priorHigh?: BetaParams; priorLow?: BetaParams }>;
|
|
156
|
+
defaultWeight?: number; // default 1
|
|
157
|
+
defaultPriorHigh?: BetaParams; // default Beta(2, 1)
|
|
158
|
+
defaultPriorLow?: BetaParams; // default Beta(1, 2)
|
|
159
|
+
});
|
|
160
|
+
|
|
161
|
+
manager.observe(observation: LabeledObservation): this; // fold one labeled output
|
|
162
|
+
manager.fit(observations: Iterable<LabeledObservation>): this;
|
|
163
|
+
manager.models(): DimensionModel[]; // snapshot for the Bayes Factor
|
|
164
|
+
manager.dimensions: string[];
|
|
165
|
+
manager.observationCount(dimension: string, kind: HypothesisKind): number;
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
#### Policy (`./policy`)
|
|
169
|
+
|
|
170
|
+
```ts
|
|
171
|
+
type PolicyConfig = BayesFactorPolicy | DecisionTheoreticPolicy;
|
|
172
|
+
|
|
173
|
+
interface BayesFactorPolicy {
|
|
174
|
+
kind: "bayes-factor";
|
|
175
|
+
passAbove: number; // pass at or above this Bayes Factor; must be >= 1
|
|
176
|
+
failBelow: number; // fail at or below this Bayes Factor; must be in (0, 1]
|
|
177
|
+
}
|
|
178
|
+
|
|
179
|
+
interface DecisionTheoreticPolicy {
|
|
180
|
+
kind: "decision-theoretic";
|
|
181
|
+
priorHighQuality: number; // base rate of high quality, in (0, 1)
|
|
182
|
+
lossFalsePass: number; // cost of passing a low-quality output, >= 0
|
|
183
|
+
lossFalseFail: number; // cost of failing a high-quality output, >= 0
|
|
184
|
+
escalationCost: number; // fixed cost of escalating to review, >= 0
|
|
185
|
+
}
|
|
186
|
+
|
|
187
|
+
decide(logBayesFactor: number, policy: PolicyConfig): PolicyDecision;
|
|
188
|
+
posteriorHighQuality(logBayesFactor: number, priorHighQuality: number): number;
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
#### Calibration (`./calibration`)
|
|
192
|
+
|
|
193
|
+
```ts
|
|
194
|
+
goodnessOfFit(samples: readonly number[], params: BetaParams, alpha?: number): GoodnessOfFit;
|
|
195
|
+
brierScore(predictions: readonly Prediction[]): number;
|
|
196
|
+
expectedCalibrationError(predictions: readonly Prediction[], bins?: number): number;
|
|
197
|
+
reliability(predictions: readonly Prediction[], bins?: number): ReliabilityBin[];
|
|
198
|
+
assessAssumptions(history, models, options?): AssumptionReport; // per-dimension fit + dependence, for the gate guards
|
|
199
|
+
|
|
200
|
+
interface GoodnessOfFit {
|
|
201
|
+
readonly ksStatistic: number;
|
|
202
|
+
readonly criticalValue: number;
|
|
203
|
+
readonly adequate: boolean; // true when the Beta assumption is reasonable
|
|
204
|
+
readonly samples: number;
|
|
205
|
+
}
|
|
206
|
+
interface Prediction { readonly probability: number; readonly outcome: number; } // outcome 0 or 1
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
#### Dependence (`./dimensions`)
|
|
210
|
+
|
|
211
|
+
```ts
|
|
212
|
+
dependenceDiagnostic(history: readonly ScoreVector[], options?: { threshold?: number }): DependenceDiagnostic;
|
|
213
|
+
|
|
214
|
+
interface DependenceDiagnostic {
|
|
215
|
+
readonly dimensions: readonly string[];
|
|
216
|
+
readonly pairs: readonly DimensionCorrelation[]; // all pairs, by absolute correlation desc
|
|
217
|
+
readonly flagged: readonly DimensionCorrelation[]; // pairs at or above the threshold
|
|
218
|
+
readonly maxAbsCorrelation: number;
|
|
219
|
+
readonly independenceAssumptionSafe: boolean; // true when no pair crosses the threshold
|
|
220
|
+
}
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
### 2.4 Error model
|
|
224
|
+
|
|
225
|
+
A single error type carries a stable, machine-readable code. Callers branch on `error.code`, never on message text.
|
|
226
|
+
|
|
227
|
+
```
|
|
228
|
+
Error
|
|
229
|
+
└─ BayesOutputGateError { code: BayesOutputGateErrorCode }
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
| Code | Raised when |
|
|
233
|
+
|---|---|
|
|
234
|
+
| `INVALID_CONFIG` | A malformed option (a bad policy kind, a negative weight, a CLI flag without a numeric value, an unreadable file) |
|
|
235
|
+
| `INVALID_SCORE` | A score outside [0, 1], a non-finite score, a duplicate dimension, a malformed score row |
|
|
236
|
+
| `INVALID_DIMENSION` | An operation referenced an unknown or invalid dimension |
|
|
237
|
+
| `INVALID_HYPOTHESIS` | An invalid hypothesis label where `high` or `low` was required |
|
|
238
|
+
| `INVALID_OBSERVATION` | A malformed labeled observation, too few samples for a diagnostic, a missing label |
|
|
239
|
+
| `INVALID_STATE` | An operation invalid for the current state, or Web Crypto unavailable |
|
|
240
|
+
| `INVALID_SNAPSHOT` | A malformed model snapshot or audit chain |
|
|
241
|
+
| `NUMERIC` | A numeric domain error, for example a Beta fit producing non-finite parameters |
|
|
242
|
+
|
|
243
|
+
### 2.5 Audit events
|
|
244
|
+
|
|
245
|
+
Each entry has `index`, `payload`, an optional caller-supplied `timestamp`, `previousHash`, and `hash`. `append` chains a SHA-256 hash over the canonicalized entry via the Web Crypto API; `verifyChain` recomputes the chain and returns `{ valid, brokenAt? }`. The seal is an integrity seal, not a digital signature: it proves a log was not altered after sealing, not who produced it. The log is tamper-evident, meaning alteration is detectable, not unalterable. Time is never read inside the library; timestamps are caller-supplied so hashing stays deterministic.
|
|
246
|
+
|
|
247
|
+
---
|
|
248
|
+
|
|
249
|
+
## 3. Architecture
|
|
250
|
+
|
|
251
|
+
```
|
|
252
|
+
+-------------------------------------------------+
|
|
253
|
+
| Caller code |
|
|
254
|
+
| const d = evaluate(scores, models, policy) |
|
|
255
|
+
| d.action // pass | fail | escalate |
|
|
256
|
+
| d.bayesFactor // the evidence ratio |
|
|
257
|
+
| d.contributions // per-dimension breakdown |
|
|
258
|
+
+-----------------------+-------------------------+
|
|
259
|
+
| per-dimension quality scores
|
|
260
|
+
v
|
|
261
|
+
+-------------------------------------------------+
|
|
262
|
+
| Gate orchestrator (gate) |
|
|
263
|
+
| - Calibrated Beta models (beta) |
|
|
264
|
+
| - Hypothesis manager (hypothesis) |
|
|
265
|
+
| - Likelihoods (likelihood) |
|
|
266
|
+
| - Bayes Factor + Jeffreys scale (bayesfactor) |
|
|
267
|
+
| - Decision policy (policy) |
|
|
268
|
+
| - Dependence diagnostic (dimensions) |
|
|
269
|
+
| - Calibration measurement (calibration) |
|
|
270
|
+
| - Tool adapter (adapter) |
|
|
271
|
+
| - Audit trail (audit) |
|
|
272
|
+
+-----------------------+-------------------------+
|
|
273
|
+
| GateDecision
|
|
274
|
+
v
|
|
275
|
+
Caller acts
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
### 3.1 The calibrated Beta model
|
|
279
|
+
|
|
280
|
+
The computational primitive is `BetaModel` (`./beta`), a Beta distribution over a score in [0, 1]. It is fit from labeled scores by method of moments, regularized by a prior Beta treated as `priorA + priorB` pseudo-observations carrying the prior's own mean and variance, and updated online from sufficient statistics (count, sum, sum of squares) so a cold model returns its prior and a warm model is data-driven. Both Bayes Factor modes reduce to this engine. A `snapshot` and `fromSnapshot` make a model serializable across processes.
|
|
281
|
+
|
|
282
|
+
### 3.2 The hypothesis manager
|
|
283
|
+
|
|
284
|
+
`HypothesisManager` (`./hypothesis`) keeps a high and a low `BetaModel` per dimension plus a weight, calibrated from `LabeledObservation`s. The weakly-informative defaults, Beta(2, 1) for high and Beta(1, 2) for low with only three pseudo-observations of concentration, let labeled data dominate quickly. `models()` snapshots the current calibration as the `DimensionModel[]` the Bayes Factor consumes.
|
|
285
|
+
|
|
286
|
+
### 3.3 Likelihoods
|
|
287
|
+
|
|
288
|
+
`./likelihood` provides both closed-form modes: the calibrated Beta log-density of a continuous score under a hypothesis (`logScoreLikelihood`), and the Beta-Binomial marginal likelihood of binary pass/fail outcomes under a hypothesis's Beta prior (`logBinaryMarginalLikelihood`). `logVectorLikelihood` sums the weighted per-dimension log-density of a score vector under one hypothesis and reports how many dimensions matched, so a caller can detect an output that scored no modeled dimension.
|
|
289
|
+
|
|
290
|
+
### 3.4 The Bayes Factor
|
|
291
|
+
|
|
292
|
+
`bayesFactor` (`./bayesfactor`) computes the combined natural-log Bayes Factor as the weighted sum of per-dimension differences between the high-hypothesis and low-hypothesis Beta log-densities, over the dimensions that have both a score and a model. Boundary scores are clamped inward by a small epsilon so a 0 or 1 never produces a non-finite log-density. `jeffreysStrength` maps the log-Bayes-Factor to the seven-way Jeffreys-scale category using the symmetric boundaries 3, 10, and 100.
|
|
293
|
+
|
|
294
|
+
### 3.5 The decision policy
|
|
295
|
+
|
|
296
|
+
`decide` (`./policy`) applies a policy to the log-Bayes-Factor. The Bayes Factor policy passes at or above `passAbove`, fails at or below `failBelow`, and escalates in between. The decision-theoretic policy converts the log-Bayes-Factor to a posterior probability of high quality through an explicit prior (`posteriorHighQuality`, via posterior log-odds equal log-Bayes-Factor plus prior log-odds, with a numerically stable logistic), then chooses the action minimizing expected loss under the asymmetric costs of a false pass, a false fail, and escalation.
|
|
297
|
+
|
|
298
|
+
### 3.6 The dependence diagnostic
|
|
299
|
+
|
|
300
|
+
`dependenceDiagnostic` (`./dimensions`) measures pairwise Pearson correlation between dimensions across a history of score vectors, over the rows where both dimensions are present, and flags pairs at or above a threshold (default 0.5). The combined Bayes Factor assumes the dimensions are conditionally independent given the hypothesis; a flagged pair warns that correlated dimensions will double-count evidence and inflate the Bayes Factor. The diagnostic is informational; the caller decides whether to drop, merge, or down-weight a flagged dimension.
|
|
301
|
+
|
|
302
|
+
### 3.7 Calibration measurement
|
|
303
|
+
|
|
304
|
+
`./calibration` measures trust in two layers, in order. First, `goodnessOfFit` tests whether the per-dimension Beta assumption holds, by the Kolmogorov-Smirnov statistic against the fitted Beta with an asymptotic critical value; when it fails, the density-ratio likelihood should not be trusted for those scores. Second, once the gate emits posterior probabilities, `brierScore`, `expectedCalibrationError`, and `reliability` measure whether those probabilities match realized outcomes. Calibration is measured, never asserted.
|
|
305
|
+
|
|
306
|
+
### 3.8 The gate orchestrator
|
|
307
|
+
|
|
308
|
+
`evaluate` (`./gate`) ties the calibrated models, the Bayes Factor, and the policy into one call. `OutputGate` wraps fixed models and a fixed policy. `OutputGateMonitor` wraps the same pipeline with a tamper-evident audit chain and online recalibration: `record` evaluates and seals each decision, `observe` and `fit` fold labeled outcomes back into the calibration, and `auditTrail` returns the sealed chain.
|
|
309
|
+
|
|
310
|
+
### 3.9 The tool adapter
|
|
311
|
+
|
|
312
|
+
`bayesOutputGateTool` (`./adapter`) exposes the whole gate as a framework-agnostic tool: a name (`bayes_output_gate`), a description, a JSON Schema for the input (scores, models, optional policy), and a handler. The shape matches what MCP servers and LLM tool-calling APIs expect. Input arriving from a model is validated defensively, and the output is made JSON-safe (non-finite numbers become null) so it survives serialization across a tool boundary.
|
|
313
|
+
|
|
314
|
+
### 3.10 Audit trail
|
|
315
|
+
|
|
316
|
+
`AuditChain` (`./audit`) records decisions append-only. `append` chains a SHA-256 hash over the canonicalized entry via the Web Crypto API; `verifyChain` recomputes and reports the first broken index; `sha256Hex` is the underlying digest. Canonical JSON serialization (key-sorted) makes the chain reproducible across runtimes. The log is tamper-evident: any alteration of a past entry breaks the chain and is detected.
|
|
317
|
+
|
|
318
|
+
---
|
|
319
|
+
|
|
320
|
+
## 4. Operational SLOs
|
|
321
|
+
|
|
322
|
+
The library is small; targets here are runtime characteristics, not service SLOs.
|
|
323
|
+
|
|
324
|
+
| Target | Budget |
|
|
325
|
+
|---|---|
|
|
326
|
+
| Runtime dependencies (required) | 0 |
|
|
327
|
+
| ESM core facade (`import { evaluate }`) | <= 8 KB brotli |
|
|
328
|
+
| Bayes Factor | exact closed-form Beta log-density ratio, summed in log space |
|
|
329
|
+
| Calibration | measured by goodness-of-fit, Brier score, and ECE, not asserted |
|
|
330
|
+
| Engines | Node >= 20.0.0, tested on 20, 22, and 24 |
|
|
331
|
+
| Node-free core | the audit seal uses Web Crypto, not node:crypto |
|
|
332
|
+
|
|
333
|
+
Bundle budgets are enforced by `size-limit` in CI.
|
|
334
|
+
|
|
335
|
+
Optional peers `@takk/keymesh` and `@takk/modelchain` are not required to run; they are integration points and the core takes no dependency on them.
|
|
336
|
+
|
|
337
|
+
---
|
|
338
|
+
|
|
339
|
+
## 5. Stability promise
|
|
340
|
+
|
|
341
|
+
### 5.1 What counts as the public API
|
|
342
|
+
|
|
343
|
+
For 1.0.0 onward:
|
|
344
|
+
|
|
345
|
+
- Every name exported from the root and from each subpath export.
|
|
346
|
+
- Every type, interface, class shape, function signature, and discriminated-union variant reachable from those exports.
|
|
347
|
+
- The shape of `QualityScore`, `ScoreVector`, `DimensionModel`, `LabeledObservation`, `BayesFactorResult`, `GateDecision`, `PolicyConfig`, the calibration types, and every audit type.
|
|
348
|
+
- The CLI flags and subcommands of `bayesoutputgate` and its exit codes.
|
|
349
|
+
- The JSON schema of the tool input and of an audit chain.
|
|
350
|
+
|
|
351
|
+
Not part of the public API:
|
|
352
|
+
|
|
353
|
+
- Anything inside `src/` that is not re-exported from a public entry point.
|
|
354
|
+
- The internal numerics of the models (the special functions, the Beta fit regularization constants).
|
|
355
|
+
- The format of any debug output.
|
|
356
|
+
|
|
357
|
+
The adapter tool (`bayesOutputGateTool`) and its JSON Schema are stable, but the integration with a specific external runtime is the consumer's responsibility; the gate returns a recommendation and does not act.
|
|
358
|
+
|
|
359
|
+
### 5.2 SemVer policy
|
|
360
|
+
|
|
361
|
+
First cut is 1.0.0.
|
|
362
|
+
|
|
363
|
+
| Change | Bump |
|
|
364
|
+
|---|---|
|
|
365
|
+
| Bug fix, internal refactor, doc-only | patch (`1.0.0 -> 1.0.1`) |
|
|
366
|
+
| New optional export, new optional field, new policy, new entry point | minor (`1.0.0 -> 1.1.0`) |
|
|
367
|
+
| Removing or renaming an export, a signature change, an audit-chain schema change, a CLI flag removal | major (`1.0.0 -> 2.0.0`), only after a deprecation cycle |
|
|
368
|
+
|
|
369
|
+
Prerelease channels are `alpha`, `beta`, and `rc`, published under the matching npm dist-tag.
|
|
370
|
+
|
|
371
|
+
### 5.3 Deprecation policy
|
|
372
|
+
|
|
373
|
+
Breaking a public API requires:
|
|
374
|
+
|
|
375
|
+
1. **Announce** the deprecation in a minor release of the current major: add `@deprecated` JSDoc on the export.
|
|
376
|
+
2. **Ship** the deprecated API for at least one further minor of the same major. Consumers must always have a non-deprecated path.
|
|
377
|
+
3. **Remove** only in the next major release, accompanied by a `MIGRATING.md` with a migration recipe.
|
|
378
|
+
|
|
379
|
+
Security-driven exceptions ship in the next patch across all supported majors with a `### Security` CHANGELOG entry. Report security issues to davcavalcante@proton.me.
|
|
380
|
+
|
|
381
|
+
### 5.4 License and provenance invariants
|
|
382
|
+
|
|
383
|
+
- License stays Apache-2.0 within a major.
|
|
384
|
+
- `NOTICE` is preserved verbatim in the tarball.
|
|
385
|
+
- Every release is published with npm provenance (`--provenance`, an OIDC SLSA attestation by GitHub Actions). Consumers can verify via `npm view @takk/bayesoutputgate@<version> --json | jq .dist.attestations`.
|
|
386
|
+
|
|
387
|
+
---
|
|
388
|
+
|
|
389
|
+
## 6. CLI
|
|
390
|
+
|
|
391
|
+
The `bayesoutputgate` binary exposes four subcommands. History is a JSON array of labeled observations (`{ scores, label }`); scores is a JSON array of score vectors (each an array of `{ dimension, value }` or an object map), or a CSV with a header row of dimension names and one row of values per output.
|
|
392
|
+
|
|
393
|
+
### 6.1 `gate <history.json> <scores.json>`
|
|
394
|
+
|
|
395
|
+
Fits the two hypotheses from labeled history, then decides each output's scores under a Bayes Factor policy, printing the action, the Bayes Factor, and the strength per output.
|
|
396
|
+
|
|
397
|
+
### 6.2 `bayes-factor <models.json> <scores.json>`
|
|
398
|
+
|
|
399
|
+
Computes the Bayes Factor for each output against an explicit array of dimension models, printing the Bayes Factor, the strength, and the favored hypothesis per output.
|
|
400
|
+
|
|
401
|
+
### 6.3 `calibrate <history.json>`
|
|
402
|
+
|
|
403
|
+
Fits from labeled history, then reports the goodness-of-fit per dimension for both hypotheses and the dependence diagnostic across dimensions.
|
|
404
|
+
|
|
405
|
+
### 6.4 `audit-verify <chain.json>`
|
|
406
|
+
|
|
407
|
+
Verifies the integrity of a sealed decision audit-chain JSON file.
|
|
408
|
+
|
|
409
|
+
| Flag | Meaning |
|
|
410
|
+
|---|---|
|
|
411
|
+
| `--pass-above` | Bayes Factor pass threshold for `gate` (default 10) |
|
|
412
|
+
| `--fail-below` | Bayes Factor fail threshold for `gate` (default 0.1) |
|
|
413
|
+
| `--json` | Emit machine-readable JSON instead of text |
|
|
414
|
+
| `--help`, `-h` | Show help |
|
|
415
|
+
| `--version`, `-v` | Show the version |
|
|
416
|
+
|
|
417
|
+
### 6.5 Exit codes
|
|
418
|
+
|
|
419
|
+
| Code | Meaning |
|
|
420
|
+
|---|---|
|
|
421
|
+
| `0` | Success (all outputs passed, or an informational command) |
|
|
422
|
+
| `2` | Usage or input error (bad flags, missing file, malformed input) |
|
|
423
|
+
| `20` | A broken audit chain (`audit-verify`) |
|
|
424
|
+
| `30` | A fail decision (`gate`) |
|
|
425
|
+
| `40` | An escalate decision (`gate`) |
|
|
426
|
+
|
|
427
|
+
---
|
|
428
|
+
|
|
429
|
+
## 7. Runtime expectations
|
|
430
|
+
|
|
431
|
+
- BayesOutputGate is a library; it does not call out to any service at import time and makes no outbound network calls of its own.
|
|
432
|
+
- The Bayes Factor is exact in closed form given the calibrated Beta models. The fit is deterministic given the same data.
|
|
433
|
+
- All stochastic routines (the posterior-predictive sample helpers) are driven by a seedable generator, so any sampling is reproducible and therefore auditable.
|
|
434
|
+
- The gate produces a pass, fail, or escalate recommendation; it does not act on your outputs. Your orchestrator consumes the decision and acts.
|
|
435
|
+
- Decisions are calibrated GIVEN the model. The numbers are exactly as trustworthy as the Beta family fits the true score distribution; goodness-of-fit and the dependence diagnostic are provided so the caller can detect when the assumptions break, and decision calibration is measured rather than assumed.
|
|
436
|
+
- The only asynchronous surfaces are the audit append and verify, which await the Web Crypto digest, and the file loaders, which await disk reads.
|
|
437
|
+
|
|
438
|
+
---
|
|
439
|
+
|
|
440
|
+
## 8. Test surface
|
|
441
|
+
|
|
442
|
+
- Unit tests for every module: the special functions, the random generators, the canonical serializer, the calibrated Beta model, the hypothesis manager, the likelihoods, the Bayes Factor, the dependence diagnostic, the decision policy, the calibration measurement, the gate, the audit chain, the tool adapter, the node loaders, and the CLI.
|
|
443
|
+
- Oracle tests that verify the Beta log-density and Beta-Binomial marginal against closed forms, the Bayes Factor against the Jeffreys boundaries, and the decision-theoretic policy against expected loss.
|
|
444
|
+
- Audit tests that confirm a known SHA-256 digest, that chains link, and that tampering and broken links are detected.
|
|
445
|
+
- Assumption-guard tests that confirm the gate escalates on an inadequate goodness-of-fit or an unsafe dependence and attaches the report to the decision.
|
|
446
|
+
- A distribution smoke test that exercises the compiled ESM and CJS artifacts and spawns the compiled CLI as a single Node process.
|
|
447
|
+
- Five runnable examples and a value benchmark under `examples/` and `benchmarks/`, executed against the compiled `dist`; every number in the docs comes from real execution.
|
|
448
|
+
|
|
449
|
+
Coverage thresholds enforced via `vitest.config.ts`: `lines >= 80`, `functions >= 80`, `statements >= 80`, `branches >= 80`. The measured coverage of 1.0.0 is statements 94.64%, branches 87.58%, functions 96.24%, lines 95.91%, over 90 tests in 15 suites.
|
|
450
|
+
|
|
451
|
+
---
|
|
452
|
+
|
|
453
|
+
## 9. Non-goals (in 1.0)
|
|
454
|
+
|
|
455
|
+
- Per-output, per-dimension automatic gating (1.0 assesses goodness-of-fit and dependence over the calibration history and escalates the whole gate via opt-in guards; a finer-grained per-output adequacy signal is a planned addition).
|
|
456
|
+
- A built-in scorer or judge (1.0 consumes scores from any scorer you already run; it does not produce them).
|
|
457
|
+
- Streaming, multi-rater aggregation, or inter-annotator agreement (planned for a later release).
|
|
458
|
+
- Drift and change-point detection on the calibrated models (planned for a later release).
|
|
459
|
+
- Non-Beta score families, for example a calibrated mixture or a nonparametric density (the 1.0 likelihood is Beta; planned for a later release).
|
|
460
|
+
- Turnkey runtime integrations that act on the decision (the 1.0 gate recommends; your orchestrator acts; bindings planned for a later release).
|
|
461
|
+
- Signed and timestamped audit seals (the current seal is integrity-only and tamper-evident, not a signature).
|
|
462
|
+
- A hosted validation dashboard and federated calibration sharing (separate products).
|
|
463
|
+
|
|
464
|
+
---
|
|
465
|
+
|
|
466
|
+
**Owner:** David C Cavalcante, davcavalcante@proton.me, Takk Innovate Studio, https://takk.ag
|
|
467
|
+
**Repository:** github.com/davccavalcante/bayesoutputgate
|