@artemiskit/sdk 0.6.3 → 0.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +33 -0
- package/dist/__tests__/experiment-fixtures.d.ts +44 -0
- package/dist/__tests__/experiment-fixtures.d.ts.map +1 -0
- package/dist/builders/index.js +19 -66
- package/dist/contracts/index.js +3 -50
- package/dist/experiment-files.d.ts +54 -0
- package/dist/experiment-files.d.ts.map +1 -0
- package/dist/experiment-fixture.d.ts +20 -0
- package/dist/experiment-fixture.d.ts.map +1 -0
- package/dist/experiment-summary.d.ts +44 -0
- package/dist/experiment-summary.d.ts.map +1 -0
- package/dist/experiment-validation.d.ts +48 -0
- package/dist/experiment-validation.d.ts.map +1 -0
- package/dist/experiment.d.ts +162 -0
- package/dist/experiment.d.ts.map +1 -0
- package/dist/guardian/circuit-breaker.d.ts +6 -1
- package/dist/guardian/circuit-breaker.d.ts.map +1 -1
- package/dist/index.d.ts +5 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +2557 -1863
- package/dist/matchers/index.js +13 -60
- package/dist/matchers/jest.js +0 -47
- package/dist/matchers/vitest.js +0 -47
- package/dist/utils/index.js +11 -58
- package/package.json +6 -5
- package/src/__tests__/artemiskit.test.ts +1 -2
- package/src/__tests__/builders.test.ts +0 -2
- package/src/__tests__/circuit-breaker.test.ts +1 -1
- package/src/__tests__/compare.test.ts +1 -1
- package/src/__tests__/contracts.test.ts +1 -1
- package/src/__tests__/experiment-case-level.test.ts +212 -0
- package/src/__tests__/experiment-fixtures.ts +194 -0
- package/src/__tests__/experiment.test.ts +1262 -0
- package/src/__tests__/guardian-modes.test.ts +2 -2
- package/src/__tests__/guardrails.test.ts +1 -1
- package/src/__tests__/matchers.test.ts +1 -1
- package/src/__tests__/semantic-validator.test.ts +15 -10
- package/src/__tests__/validate.test.ts +0 -4
- package/src/__tests__/workflow-recovery.test.ts +7 -3
- package/src/__tests__/workflow-report.test.ts +6 -3
- package/src/__tests__/workflow.test.ts +11 -5
- package/src/experiment-files.ts +190 -0
- package/src/experiment-fixture.ts +70 -0
- package/src/experiment-summary.ts +172 -0
- package/src/experiment-validation.ts +426 -0
- package/src/experiment.ts +630 -0
- package/src/guardian/circuit-breaker.ts +6 -1
- package/src/index.ts +52 -0
- package/tsconfig.typecheck.json +12 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,38 @@
|
|
|
1
1
|
# @artemiskit/sdk
|
|
2
2
|
|
|
3
|
+
## 0.7.1
|
|
4
|
+
|
|
5
|
+
### Patch Changes
|
|
6
|
+
|
|
7
|
+
- Updated dependencies [56e134e]
|
|
8
|
+
- @artemiskit/core@0.7.1
|
|
9
|
+
- @artemiskit/redteam@0.3.14
|
|
10
|
+
- @artemiskit/reports@0.6.1
|
|
11
|
+
|
|
12
|
+
## 0.7.0
|
|
13
|
+
|
|
14
|
+
### Minor Changes
|
|
15
|
+
|
|
16
|
+
- 4882eff: Experiment reports now show a case-level summary beside the coordinate-level summary. A scenario coordinate (task × target × repetition) passes only when all of its cases pass, so coordinate counts alone understate how many individual cases passed. The case level counts planned, recorded, valid, passed, failed, invalid and unmeasured cases overall and per target, task, language and policy, read from each coordinate's terminal attempt evidence. Each saved evidence file must hash to the SHA-256 recorded in the result, parse as a run manifest and agree with the saved coordinate status; otherwise the whole case level is withheld with its reason instead of showing partial counts. Agent-workflow coordinates have no scenario cases and are counted separately. Results saved without an evidence directory still report, with case level marked unavailable.
|
|
17
|
+
|
|
18
|
+
- `@artemiskit/reports`: `createExperimentReport` and `generateExperimentReport` accept `caseEvidence` (evidence bytes keyed by recorded digest) and add `caseLevel` to the report; new `createExperimentCaseSummary` and `listExperimentCaseEvidenceDigests`. Technical, executive and comprehensive views (HTML and Markdown) label both levels and make no ranking claim; technical and comprehensive views add a per-task, language and policy table.
|
|
19
|
+
- `@artemiskit/sdk`: new `summarizeExperimentCases` and `formatExperimentCaseSummary`; `reportExperimentResult` reads `evidence/` beside a result file or an explicit `evidenceDir`; `runExperimentFile` returns `caseLevel`; `formatExperimentRunSummary` accepts `caseLevel`.
|
|
20
|
+
- `@artemiskit/cli`: `akit experiment run` and `akit experiment report` print the case level after the coordinate counts; `report` gains `--evidence-dir`.
|
|
21
|
+
|
|
22
|
+
- f43d5f2: Add the experiment surfaces. The SDK exports `validateExperimentFile`, `runExperimentFile`, `reportExperimentResult`, `createExperimentLiveAuthorization` and `createExperimentFixtureClient`, with exit codes shared with the CLI. The CLI adds `akit experiment validate`, `run` and `report`. Validation checks the manifest, the SHA-256 of each task source and that every cost-not-applicable target uses a loopback endpoint. `scenario_evaluation` tasks run through the scenario runner and `agent_workflow` tasks through the native workflow engine, both bound to the same targets, with each target's generation settings and the per-repetition seed applied to both kinds; validation parses workflow sources and warns about workflows with semantic outcomes, which experiment runs can't judge. Runs use deterministic fixture clients by default with no network calls. A live run requires `--approve`, `--approved-by` and `--reason`, and `--max-concurrency` for a declared concurrency above 1, before anything executes. Loopback targets get a non-secret placeholder key instead of an ambient API key. Provider configuration is loaded only when a live non-loopback target is bound, and target config resolvers may be async. Live adapters get a timeout of at least the per-case timeout. Results and digest-named evidence are saved under `artemis-output/experiments/`, and validation and run output show the concurrency level. `report` regenerates all three views in HTML and Markdown from a saved result without provider access. The SDK now declares its `yaml` dependency, and the CLI depends on the SDK.
|
|
23
|
+
|
|
24
|
+
### Patch Changes
|
|
25
|
+
|
|
26
|
+
- cde9742: `new CircuitBreaker(config)` now types its parameter as `Partial<CircuitBreakerConfig>`, matching what it already did at runtime: omitted fields default to `enabled: true`, `threshold: 5`, `windowMs: 60000`, `cooldownMs: 300000` and `halfOpenRequests: 3`. Type-only change; behaviour is unchanged and full configs still type-check.
|
|
27
|
+
- Updated dependencies [4b3a6a9]
|
|
28
|
+
- Updated dependencies [4882eff]
|
|
29
|
+
- Updated dependencies [f20ee54]
|
|
30
|
+
- Updated dependencies [515a69f]
|
|
31
|
+
- Updated dependencies [ae38f94]
|
|
32
|
+
- @artemiskit/core@0.7.0
|
|
33
|
+
- @artemiskit/reports@0.6.0
|
|
34
|
+
- @artemiskit/redteam@0.3.13
|
|
35
|
+
|
|
3
36
|
## 0.6.3
|
|
4
37
|
|
|
5
38
|
### Patch Changes
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Test helper: write a self-contained two-target experiment directory.
|
|
3
|
+
* Shared by the SDK and CLI experiment tests.
|
|
4
|
+
*/
|
|
5
|
+
/** Passes against the fixture client, whose responses start with "Fixture response". */
|
|
6
|
+
export declare const PASSING_SCENARIO = "name: fixture-pass\ncases:\n - id: echo\n prompt: \"Reply with anything.\"\n expected:\n type: contains\n values: [\"Fixture response\"]\n";
|
|
7
|
+
/** A valid measurement that fails against the fixture client. */
|
|
8
|
+
export declare const FAILING_SCENARIO = "name: fixture-fail\ncases:\n - id: capital\n prompt: \"What is the capital of Nigeria?\"\n expected:\n type: contains\n values: [\"Abuja\"]\n";
|
|
9
|
+
/**
|
|
10
|
+
* Agent workflow that passes against the fixture client, which accepts tools but never calls
|
|
11
|
+
* one: the target must answer without reading the brief and without any undeclared tool.
|
|
12
|
+
*/
|
|
13
|
+
export declare const FIXTURE_WORKFLOW = "version: '1'\nkind: agent_workflow\nname: fixture-workflow\ntarget:\n provider: openai\n model: declared-model\nenvironment:\n type: simulated\n policy:\n network: denied\n side_effects: denied\n permissions:\n documents: read\n budgets:\n max_actions: 4\n max_model_requests: 2\n timeout_ms: 10000\ntools:\n - read_document\nworkflow:\n system_instructions: Answer from the conversation; read the brief only when asked to.\n initial_state:\n documents:\n brief: Fixture brief.\n turns:\n - role: user\n content: Reply with anything.\noutcomes:\n deterministic:\n - type: tool_trace\n tool: read_document\n minimum_calls: 0\n maximum_calls: 0\n - type: policy\n rule: no_undeclared_tool\n expected: passed\nevidence:\n trace: summary\n artifacts: checksums\n redact: true\n";
|
|
14
|
+
export declare const LOOPBACK_BASE_URL = "http://127.0.0.1:11434/v1";
|
|
15
|
+
export interface ExperimentFixtureOptions {
|
|
16
|
+
/** Override manifest fields after the defaults are built. */
|
|
17
|
+
overrides?: Record<string, unknown>;
|
|
18
|
+
/** Replace a task's declared digest. */
|
|
19
|
+
digestOverride?: Record<string, string>;
|
|
20
|
+
/** Skip writing these source files. */
|
|
21
|
+
omitSources?: string[];
|
|
22
|
+
/** Target transforms applied to both default targets. */
|
|
23
|
+
target?: (target: Record<string, unknown>) => Record<string, unknown>;
|
|
24
|
+
/** Manifest file name (default `experiment.yaml`; the content is JSON, which YAML accepts). */
|
|
25
|
+
fileName?: string;
|
|
26
|
+
/**
|
|
27
|
+
* Declare one scenario task (`pass`) and one agent workflow task (`workflow`) instead of the
|
|
28
|
+
* two scenario tasks.
|
|
29
|
+
*/
|
|
30
|
+
workflow?: boolean;
|
|
31
|
+
}
|
|
32
|
+
export interface ExperimentFixture {
|
|
33
|
+
directory: string;
|
|
34
|
+
file: string;
|
|
35
|
+
manifest: Record<string, unknown>;
|
|
36
|
+
}
|
|
37
|
+
export declare function sha256(text: string): string;
|
|
38
|
+
/** Live controls every live manifest must declare. */
|
|
39
|
+
export declare const LIVE_CONTROLS: {
|
|
40
|
+
concurrency_ceiling: number;
|
|
41
|
+
stop_conditions: string[];
|
|
42
|
+
};
|
|
43
|
+
export declare function writeExperimentFixture(options?: ExperimentFixtureOptions): Promise<ExperimentFixture>;
|
|
44
|
+
//# sourceMappingURL=experiment-fixtures.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"experiment-fixtures.d.ts","sourceRoot":"","sources":["../../src/__tests__/experiment-fixtures.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAOH,wFAAwF;AACxF,eAAO,MAAM,gBAAgB,gKAO5B,CAAC;AAEF,iEAAiE;AACjE,eAAO,MAAM,gBAAgB,mKAO5B,CAAC;AAEF;;;GAGG;AACH,eAAO,MAAM,gBAAgB,q2BAwC5B,CAAC;AAEF,eAAO,MAAM,iBAAiB,8BAA8B,CAAC;AAE7D,MAAM,WAAW,wBAAwB;IACvC,6DAA6D;IAC7D,SAAS,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;IACpC,wCAAwC;IACxC,cAAc,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACxC,uCAAuC;IACvC,WAAW,CAAC,EAAE,MAAM,EAAE,CAAC;IACvB,yDAAyD;IACzD,MAAM,CAAC,EAAE,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,KAAK,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;IACtE,+FAA+F;IAC/F,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB;;;OAGG;IACH,QAAQ,CAAC,EAAE,OAAO,CAAC;CACpB;AAED,MAAM,WAAW,iBAAiB;IAChC,SAAS,EAAE,MAAM,CAAC;IAClB,IAAI,EAAE,MAAM,CAAC;IACb,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;CACnC;AAQD,wBAAgB,MAAM,CAAC,IAAI,EAAE,MAAM,GAAG,MAAM,CAE3C;AAeD,sDAAsD;AACtD,eAAO,MAAM,aAAa;;;CAUzB,CAAC;AAEF,wBAAsB,sBAAsB,CAC1C,OAAO,GAAE,wBAA6B,GACrC,OAAO,CAAC,iBAAiB,CAAC,CAqD5B"}
|
package/dist/builders/index.js
CHANGED
|
@@ -1,50 +1,3 @@
|
|
|
1
|
-
import { createRequire } from "node:module";
|
|
2
|
-
var __create = Object.create;
|
|
3
|
-
var __getProtoOf = Object.getPrototypeOf;
|
|
4
|
-
var __defProp = Object.defineProperty;
|
|
5
|
-
var __getOwnPropNames = Object.getOwnPropertyNames;
|
|
6
|
-
var __hasOwnProp = Object.prototype.hasOwnProperty;
|
|
7
|
-
function __accessProp(key) {
|
|
8
|
-
return this[key];
|
|
9
|
-
}
|
|
10
|
-
var __toESMCache_node;
|
|
11
|
-
var __toESMCache_esm;
|
|
12
|
-
var __toESM = (mod, isNodeMode, target) => {
|
|
13
|
-
var canCache = mod != null && typeof mod === "object";
|
|
14
|
-
if (canCache) {
|
|
15
|
-
var cache = isNodeMode ? __toESMCache_node ??= new WeakMap : __toESMCache_esm ??= new WeakMap;
|
|
16
|
-
var cached = cache.get(mod);
|
|
17
|
-
if (cached)
|
|
18
|
-
return cached;
|
|
19
|
-
}
|
|
20
|
-
target = mod != null ? __create(__getProtoOf(mod)) : {};
|
|
21
|
-
const to = isNodeMode || !mod || !mod.__esModule ? __defProp(target, "default", { value: mod, enumerable: true }) : target;
|
|
22
|
-
for (let key of __getOwnPropNames(mod))
|
|
23
|
-
if (!__hasOwnProp.call(to, key))
|
|
24
|
-
__defProp(to, key, {
|
|
25
|
-
get: __accessProp.bind(mod, key),
|
|
26
|
-
enumerable: true
|
|
27
|
-
});
|
|
28
|
-
if (canCache)
|
|
29
|
-
cache.set(mod, to);
|
|
30
|
-
return to;
|
|
31
|
-
};
|
|
32
|
-
var __commonJS = (cb, mod) => () => (mod || cb((mod = { exports: {} }).exports, mod), mod.exports);
|
|
33
|
-
var __returnValue = (v) => v;
|
|
34
|
-
function __exportSetter(name, newValue) {
|
|
35
|
-
this[name] = __returnValue.bind(null, newValue);
|
|
36
|
-
}
|
|
37
|
-
var __export = (target, all) => {
|
|
38
|
-
for (var name in all)
|
|
39
|
-
__defProp(target, name, {
|
|
40
|
-
get: all[name],
|
|
41
|
-
enumerable: true,
|
|
42
|
-
configurable: true,
|
|
43
|
-
set: __exportSetter.bind(all, name)
|
|
44
|
-
});
|
|
45
|
-
};
|
|
46
|
-
var __require = /* @__PURE__ */ createRequire(import.meta.url);
|
|
47
|
-
|
|
48
1
|
// src/builders/index.ts
|
|
49
2
|
class TestCaseBuilder {
|
|
50
3
|
_id;
|
|
@@ -377,24 +330,24 @@ function anyOf(...expectations) {
|
|
|
377
330
|
return { type: "combined", operator: "or", expectations };
|
|
378
331
|
}
|
|
379
332
|
export {
|
|
380
|
-
|
|
381
|
-
similarity,
|
|
382
|
-
scenario,
|
|
383
|
-
regexCase,
|
|
384
|
-
regex,
|
|
385
|
-
notContains,
|
|
386
|
-
llmGrade,
|
|
387
|
-
jsonSchema,
|
|
388
|
-
jsonCase,
|
|
389
|
-
inline,
|
|
390
|
-
gradedCase,
|
|
391
|
-
fuzzy,
|
|
392
|
-
exactCase,
|
|
393
|
-
exact,
|
|
394
|
-
containsCase,
|
|
395
|
-
contains,
|
|
396
|
-
anyOf,
|
|
397
|
-
allOf,
|
|
333
|
+
ScenarioBuilder,
|
|
398
334
|
TestCaseBuilder,
|
|
399
|
-
|
|
335
|
+
allOf,
|
|
336
|
+
anyOf,
|
|
337
|
+
contains,
|
|
338
|
+
containsCase,
|
|
339
|
+
exact,
|
|
340
|
+
exactCase,
|
|
341
|
+
fuzzy,
|
|
342
|
+
gradedCase,
|
|
343
|
+
inline,
|
|
344
|
+
jsonCase,
|
|
345
|
+
jsonSchema,
|
|
346
|
+
llmGrade,
|
|
347
|
+
notContains,
|
|
348
|
+
regex,
|
|
349
|
+
regexCase,
|
|
350
|
+
scenario,
|
|
351
|
+
similarity,
|
|
352
|
+
testCase
|
|
400
353
|
};
|
package/dist/contracts/index.js
CHANGED
|
@@ -1,50 +1,3 @@
|
|
|
1
|
-
import { createRequire } from "node:module";
|
|
2
|
-
var __create = Object.create;
|
|
3
|
-
var __getProtoOf = Object.getPrototypeOf;
|
|
4
|
-
var __defProp = Object.defineProperty;
|
|
5
|
-
var __getOwnPropNames = Object.getOwnPropertyNames;
|
|
6
|
-
var __hasOwnProp = Object.prototype.hasOwnProperty;
|
|
7
|
-
function __accessProp(key) {
|
|
8
|
-
return this[key];
|
|
9
|
-
}
|
|
10
|
-
var __toESMCache_node;
|
|
11
|
-
var __toESMCache_esm;
|
|
12
|
-
var __toESM = (mod, isNodeMode, target) => {
|
|
13
|
-
var canCache = mod != null && typeof mod === "object";
|
|
14
|
-
if (canCache) {
|
|
15
|
-
var cache = isNodeMode ? __toESMCache_node ??= new WeakMap : __toESMCache_esm ??= new WeakMap;
|
|
16
|
-
var cached = cache.get(mod);
|
|
17
|
-
if (cached)
|
|
18
|
-
return cached;
|
|
19
|
-
}
|
|
20
|
-
target = mod != null ? __create(__getProtoOf(mod)) : {};
|
|
21
|
-
const to = isNodeMode || !mod || !mod.__esModule ? __defProp(target, "default", { value: mod, enumerable: true }) : target;
|
|
22
|
-
for (let key of __getOwnPropNames(mod))
|
|
23
|
-
if (!__hasOwnProp.call(to, key))
|
|
24
|
-
__defProp(to, key, {
|
|
25
|
-
get: __accessProp.bind(mod, key),
|
|
26
|
-
enumerable: true
|
|
27
|
-
});
|
|
28
|
-
if (canCache)
|
|
29
|
-
cache.set(mod, to);
|
|
30
|
-
return to;
|
|
31
|
-
};
|
|
32
|
-
var __commonJS = (cb, mod) => () => (mod || cb((mod = { exports: {} }).exports, mod), mod.exports);
|
|
33
|
-
var __returnValue = (v) => v;
|
|
34
|
-
function __exportSetter(name, newValue) {
|
|
35
|
-
this[name] = __returnValue.bind(null, newValue);
|
|
36
|
-
}
|
|
37
|
-
var __export = (target, all) => {
|
|
38
|
-
for (var name in all)
|
|
39
|
-
__defProp(target, name, {
|
|
40
|
-
get: all[name],
|
|
41
|
-
enumerable: true,
|
|
42
|
-
configurable: true,
|
|
43
|
-
set: __exportSetter.bind(all, name)
|
|
44
|
-
});
|
|
45
|
-
};
|
|
46
|
-
var __require = /* @__PURE__ */ createRequire(import.meta.url);
|
|
47
|
-
|
|
48
1
|
// src/contracts/index.ts
|
|
49
2
|
function defineAdapter(adapter) {
|
|
50
3
|
if (!adapter.provider) {
|
|
@@ -92,8 +45,8 @@ function definePlugin(plugin) {
|
|
|
92
45
|
return plugin;
|
|
93
46
|
}
|
|
94
47
|
export {
|
|
95
|
-
|
|
96
|
-
definePlugin,
|
|
48
|
+
defineAdapter,
|
|
97
49
|
defineEvaluator,
|
|
98
|
-
|
|
50
|
+
definePlugin,
|
|
51
|
+
defineStorage
|
|
99
52
|
};
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* File-system boundary for experiment surfaces: bounded reads, result directories and
|
|
3
|
+
* write-once evidence files. Shared by the SDK functions and the `akit experiment` CLI.
|
|
4
|
+
*/
|
|
5
|
+
import type { ExperimentRunResult } from '@artemiskit/core';
|
|
6
|
+
/** Largest experiment definition the surfaces will read. */
|
|
7
|
+
export declare const MAX_EXPERIMENT_FILE_BYTES = 1048576;
|
|
8
|
+
/** Largest saved experiment result the report surface will read. */
|
|
9
|
+
export declare const MAX_EXPERIMENT_RESULT_BYTES: number;
|
|
10
|
+
/** Default output root, relative to the working directory; ignored by git. */
|
|
11
|
+
export declare const DEFAULT_EXPERIMENT_OUTPUT_ROOT: string;
|
|
12
|
+
export declare const EXPERIMENT_RESULT_FILE = "result.json";
|
|
13
|
+
export declare const EXPERIMENT_EVIDENCE_DIR = "evidence";
|
|
14
|
+
/**
|
|
15
|
+
* Exit-code-bearing failure for the experiment surfaces. `exitCode` is the CLI exit code the
|
|
16
|
+
* same failure maps to, so SDK callers and the CLI agree.
|
|
17
|
+
*/
|
|
18
|
+
export declare class ExperimentSurfaceError extends Error {
|
|
19
|
+
readonly code: string;
|
|
20
|
+
readonly exitCode: number;
|
|
21
|
+
constructor(code: string, message: string, exitCode: number);
|
|
22
|
+
}
|
|
23
|
+
/** Read a regular file of at most `limit` bytes as strict UTF-8. */
|
|
24
|
+
export declare function readBoundedText(file: string, limit: number): Promise<string>;
|
|
25
|
+
/** Lowercase hex SHA-256 of exact bytes. */
|
|
26
|
+
export declare function sha256Hex(bytes: Uint8Array | string): string;
|
|
27
|
+
/**
|
|
28
|
+
* Canonical JSON with recursively sorted object keys. Matches the serialization of core's
|
|
29
|
+
* `runManifestIdentity` and `workflowRecordIdentity`, so a saved evidence file's SHA-256 equals
|
|
30
|
+
* its recorded artifact digest.
|
|
31
|
+
*/
|
|
32
|
+
export declare function canonicalJson(value: unknown): string;
|
|
33
|
+
/** `artemis-output/experiments/<id>/<timestamp>` under `cwd`, with a filename-safe timestamp. */
|
|
34
|
+
export declare function defaultExperimentOutputDir(experimentId: string, cwd: string, now: Date): string;
|
|
35
|
+
/** Create `directory` (and parents) and refuse to reuse one that already holds files. */
|
|
36
|
+
export declare function prepareEmptyDirectory(directory: string): Promise<void>;
|
|
37
|
+
/** Write a new file; never overwrite. */
|
|
38
|
+
export declare function writeNewFile(file: string, content: string): Promise<void>;
|
|
39
|
+
/**
|
|
40
|
+
* Save one attempt's evidence (a scenario run manifest or a workflow record) as canonical JSON
|
|
41
|
+
* named by its digest, and prove the written bytes match the identity recorded in the
|
|
42
|
+
* experiment result. A throw marks the attempt `infrastructure_failed` in the executor.
|
|
43
|
+
*/
|
|
44
|
+
export declare function saveAttemptEvidence(directory: string, evidence: unknown, digest: string): Promise<string>;
|
|
45
|
+
/** Persist the V1 experiment result as pretty JSON. */
|
|
46
|
+
export declare function saveExperimentResult(directory: string, result: ExperimentRunResult): Promise<string>;
|
|
47
|
+
/**
|
|
48
|
+
* Read the saved evidence files named by `digests` from an `evidence/` directory, as exact bytes
|
|
49
|
+
* keyed by digest. A missing, unreadable or oversized file maps to `null` so the case-level
|
|
50
|
+
* summary reports it; the bytes are verified against the digest by the report model, not here.
|
|
51
|
+
* Returns `undefined` when the directory does not exist (results saved without evidence).
|
|
52
|
+
*/
|
|
53
|
+
export declare function readExperimentCaseEvidence(evidenceDir: string, digests: readonly string[]): Promise<Map<string, Uint8Array | null> | undefined>;
|
|
54
|
+
//# sourceMappingURL=experiment-files.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"experiment-files.d.ts","sourceRoot":"","sources":["../src/experiment-files.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAKH,OAAO,KAAK,EAAE,mBAAmB,EAAE,MAAM,kBAAkB,CAAC;AAG5D,4DAA4D;AAC5D,eAAO,MAAM,yBAAyB,UAAY,CAAC;AACnD,oEAAoE;AACpE,eAAO,MAAM,2BAA2B,QAAiB,CAAC;AAC1D,8EAA8E;AAC9E,eAAO,MAAM,8BAA8B,QAA6C,CAAC;AACzF,eAAO,MAAM,sBAAsB,gBAAgB,CAAC;AACpD,eAAO,MAAM,uBAAuB,aAAa,CAAC;AAKlD;;;GAGG;AACH,qBAAa,sBAAuB,SAAQ,KAAK;IAC/C,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;gBAEd,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM;CAM5D;AAED,oEAAoE;AACpE,wBAAsB,eAAe,CAAC,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC,CAWlF;AAED,4CAA4C;AAC5C,wBAAgB,SAAS,CAAC,KAAK,EAAE,UAAU,GAAG,MAAM,GAAG,MAAM,CAE5D;AAED;;;;GAIG;AACH,wBAAgB,aAAa,CAAC,KAAK,EAAE,OAAO,GAAG,MAAM,CAEpD;AAaD,iGAAiG;AACjG,wBAAgB,0BAA0B,CAAC,YAAY,EAAE,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,GAAG,EAAE,IAAI,GAAG,MAAM,CAG/F;AAED,yFAAyF;AACzF,wBAAsB,qBAAqB,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAW5E;AAED,yCAAyC;AACzC,wBAAsB,YAAY,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAQ/E;AAED;;;;GAIG;AACH,wBAAsB,mBAAmB,CACvC,SAAS,EAAE,MAAM,EACjB,QAAQ,EAAE,OAAO,EACjB,MAAM,EAAE,MAAM,GACb,OAAO,CAAC,MAAM,CAAC,CAcjB;AAED,uDAAuD;AACvD,wBAAsB,oBAAoB,CACxC,SAAS,EAAE,MAAM,EACjB,MAAM,EAAE,mBAAmB,GAC1B,OAAO,CAAC,MAAM,CAAC,CAYjB;AAID;;;;;GAKG;AACH,wBAAsB,0BAA0B,CAC9C,WAAW,EAAE,MAAM,EACnB,OAAO,EAAE,SAAS,MAAM,EAAE,GACzB,OAAO,CAAC,GAAG,CAAC,MAAM,EAAE,UAAU,GAAG,IAAI,CAAC,GAAG,SAAS,CAAC,CAcrD"}
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Deterministic in-memory model client for fixture-mode experiments.
|
|
3
|
+
*
|
|
4
|
+
* Fixture runs exercise the full experiment pipeline (matrix, executor, evaluators, evidence,
|
|
5
|
+
* reports) without any network access. Responses are fixed text, so fixture outcomes describe
|
|
6
|
+
* the pipeline, never a model.
|
|
7
|
+
*/
|
|
8
|
+
import type { ExperimentTarget, ModelClient } from '@artemiskit/core';
|
|
9
|
+
export declare const FIXTURE_PROVIDER = "artemiskit-fixture";
|
|
10
|
+
export interface ExperimentFixtureClientOptions {
|
|
11
|
+
/**
|
|
12
|
+
* Advertise structured tool use. The client accepts tool definitions but never calls a
|
|
13
|
+
* tool, so a fixture workflow can run (the workflow engine requires tool support) and its
|
|
14
|
+
* outcome depends only on the declared state and outcomes.
|
|
15
|
+
*/
|
|
16
|
+
toolUse?: boolean;
|
|
17
|
+
}
|
|
18
|
+
/** Create the offline client bound to one fixture-mode target. */
|
|
19
|
+
export declare function createExperimentFixtureClient(target: ExperimentTarget, options?: ExperimentFixtureClientOptions): ModelClient;
|
|
20
|
+
//# sourceMappingURL=experiment-fixture.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"experiment-fixture.d.ts","sourceRoot":"","sources":["../src/experiment-fixture.ts"],"names":[],"mappings":"AAAA;;;;;;GAMG;AAEH,OAAO,KAAK,EACV,gBAAgB,EAGhB,WAAW,EACZ,MAAM,kBAAkB,CAAC;AAG1B,eAAO,MAAM,gBAAgB,uBAAuB,CAAC;AAKrD,MAAM,WAAW,8BAA8B;IAC7C;;;;OAIG;IACH,OAAO,CAAC,EAAE,OAAO,CAAC;CACnB;AAED,kEAAkE;AAClE,wBAAgB,6BAA6B,CAC3C,MAAM,EAAE,gBAAgB,EACxB,OAAO,GAAE,8BAAmC,GAC3C,WAAW,CA2Bb"}
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Exit codes and the plain-text run summary shared by the SDK and `akit experiment run`.
|
|
3
|
+
* The summary is descriptive: it lists targets in declared order and never ranks them.
|
|
4
|
+
*/
|
|
5
|
+
import type { ExperimentRunResult, ExperimentTarget } from '@artemiskit/core';
|
|
6
|
+
import { type ExperimentCaseLevelSummary } from '@artemiskit/reports';
|
|
7
|
+
/** Exit codes for `akit experiment run`; SDK outcomes carry the same value. */
|
|
8
|
+
export declare const EXPERIMENT_EXIT_CODES: {
|
|
9
|
+
/** Completed with a valid passed or failed measurement for every declared coordinate. */
|
|
10
|
+
readonly complete: 0;
|
|
11
|
+
/** Unexpected I/O or internal failure, including an unusable output directory. */
|
|
12
|
+
readonly error: 1;
|
|
13
|
+
/** The experiment file, a task source or a target declaration failed validation. */
|
|
14
|
+
readonly invalid: 2;
|
|
15
|
+
/** Live experiment without a valid explicit approval; nothing was executed. */
|
|
16
|
+
readonly unauthorized: 3;
|
|
17
|
+
/** Execution stopped early (budget or usage stop condition); see the stop reason. */
|
|
18
|
+
readonly stopped: 4;
|
|
19
|
+
/** Finished, but some coordinates are invalid, unsupported, excluded, errored or incomplete. */
|
|
20
|
+
readonly coverageIncomplete: 5;
|
|
21
|
+
};
|
|
22
|
+
export type ExperimentRunStatus = keyof typeof EXPERIMENT_EXIT_CODES;
|
|
23
|
+
export type ExperimentExitCode = (typeof EXPERIMENT_EXIT_CODES)[ExperimentRunStatus];
|
|
24
|
+
/** Classify a finished run. Early stops take precedence over incomplete coverage. */
|
|
25
|
+
export declare function experimentRunStatus(result: ExperimentRunResult): 'complete' | 'stopped' | 'coverageIncomplete';
|
|
26
|
+
/** Deployment label for a target, e.g. "open-weight, run locally via ollama; cost not applicable". */
|
|
27
|
+
export declare function describeTargetDeployment(target: ExperimentTarget): string;
|
|
28
|
+
/**
|
|
29
|
+
* Declared concurrency limit and, for concurrent runs, the recorded peak and caveat, e.g.
|
|
30
|
+
* "up to 4 coordinates in flight (declared limit); recorded peak 3; completion order and
|
|
31
|
+
* sampled outputs may differ from a serial run".
|
|
32
|
+
*/
|
|
33
|
+
export declare function describeConcurrency(result: ExperimentRunResult): string;
|
|
34
|
+
/** Human-readable summary of a saved run. Makes no ranking or winner claim. */
|
|
35
|
+
export declare function formatExperimentRunSummary(result: ExperimentRunResult, options?: {
|
|
36
|
+
resultPath?: string;
|
|
37
|
+
caseLevel?: ExperimentCaseLevelSummary;
|
|
38
|
+
}): string;
|
|
39
|
+
/**
|
|
40
|
+
* Plain-text case-level summary: individual scenario cases from verified per-attempt evidence,
|
|
41
|
+
* per target (in `targetOrder` when given) and overall, or why it is unavailable. No ranking.
|
|
42
|
+
*/
|
|
43
|
+
export declare function formatExperimentCaseSummary(caseLevel: ExperimentCaseLevelSummary, targetOrder?: readonly string[]): string;
|
|
44
|
+
//# sourceMappingURL=experiment-summary.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"experiment-summary.d.ts","sourceRoot":"","sources":["../src/experiment-summary.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,OAAO,KAAK,EAAE,mBAAmB,EAAqB,gBAAgB,EAAE,MAAM,kBAAkB,CAAC;AACjG,OAAO,EAEL,KAAK,0BAA0B,EAEhC,MAAM,qBAAqB,CAAC;AAE7B,+EAA+E;AAC/E,eAAO,MAAM,qBAAqB;IAChC,yFAAyF;;IAEzF,kFAAkF;;IAElF,oFAAoF;;IAEpF,+EAA+E;;IAE/E,qFAAqF;;IAErF,gGAAgG;;CAExF,CAAC;AAEX,MAAM,MAAM,mBAAmB,GAAG,MAAM,OAAO,qBAAqB,CAAC;AACrE,MAAM,MAAM,kBAAkB,GAAG,CAAC,OAAO,qBAAqB,CAAC,CAAC,mBAAmB,CAAC,CAAC;AAErF,qFAAqF;AACrF,wBAAgB,mBAAmB,CACjC,MAAM,EAAE,mBAAmB,GAC1B,UAAU,GAAG,SAAS,GAAG,oBAAoB,CAI/C;AAED,sGAAsG;AACtG,wBAAgB,wBAAwB,CAAC,MAAM,EAAE,gBAAgB,GAAG,MAAM,CAOzE;AAED;;;;GAIG;AACH,wBAAgB,mBAAmB,CAAC,MAAM,EAAE,mBAAmB,GAAG,MAAM,CAKvE;AAED,+EAA+E;AAC/E,wBAAgB,0BAA0B,CACxC,MAAM,EAAE,mBAAmB,EAC3B,OAAO,GAAE;IAAE,UAAU,CAAC,EAAE,MAAM,CAAC;IAAC,SAAS,CAAC,EAAE,0BAA0B,CAAA;CAAO,GAC5E,MAAM,CAyCR;AAKD;;;GAGG;AACH,wBAAgB,2BAA2B,CACzC,SAAS,EAAE,0BAA0B,EACrC,WAAW,GAAE,SAAS,MAAM,EAAO,GAClC,MAAM,CAeR"}
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Offline validation of an experiment file: manifest contract, task source identities and the
|
|
3
|
+
* host-side locality check for targets that declare no metered cost. Makes no model calls.
|
|
4
|
+
*/
|
|
5
|
+
import { type ExperimentManifest, type ExperimentMode } from '@artemiskit/core';
|
|
6
|
+
export type ExperimentCheckSubject = 'file' | 'manifest' | 'task' | 'target';
|
|
7
|
+
export type ExperimentCheckStatus = 'ok' | 'warning' | 'error';
|
|
8
|
+
/** One validation finding. `code` is stable and machine-readable. */
|
|
9
|
+
export interface ExperimentCheck {
|
|
10
|
+
subject: ExperimentCheckSubject;
|
|
11
|
+
/** Task or target ID for task and target checks. */
|
|
12
|
+
id?: string;
|
|
13
|
+
status: ExperimentCheckStatus;
|
|
14
|
+
code: string;
|
|
15
|
+
message: string;
|
|
16
|
+
}
|
|
17
|
+
export interface ExperimentValidation {
|
|
18
|
+
/** True when no check has status `error`. Warnings do not block a run. */
|
|
19
|
+
valid: boolean;
|
|
20
|
+
/** Absolute path of the experiment file. */
|
|
21
|
+
file: string;
|
|
22
|
+
/** Directory that task source paths resolve under (the experiment file's directory). */
|
|
23
|
+
root: string;
|
|
24
|
+
experimentId?: string;
|
|
25
|
+
mode?: ExperimentMode;
|
|
26
|
+
/** Declared concurrency limit (coordinates in flight), when the manifest is valid. */
|
|
27
|
+
concurrency?: number;
|
|
28
|
+
/** Parsed manifest, present when the manifest contract itself is valid. */
|
|
29
|
+
manifest?: ExperimentManifest;
|
|
30
|
+
checks: ExperimentCheck[];
|
|
31
|
+
/** CLI exit code for `akit experiment validate`: 0 valid, 1 unreadable file, 2 invalid. */
|
|
32
|
+
exitCode: 0 | 1 | 2;
|
|
33
|
+
}
|
|
34
|
+
export declare const LOOPBACK_REQUIRED_MESSAGE = "cost not applicable is only accepted for loopback endpoints";
|
|
35
|
+
/**
|
|
36
|
+
* Validate an experiment file without contacting any model or network endpoint.
|
|
37
|
+
*
|
|
38
|
+
* Checks the manifest contract; that each task source exists under the experiment directory and
|
|
39
|
+
* its exact bytes match the declared SHA-256; that scenario and workflow sources parse; that
|
|
40
|
+
* workflow tasks can run as declared (warnings otherwise); that every target
|
|
41
|
+
* declaring `cost_basis: not_applicable` names a loopback `settings.base_url`; and that target
|
|
42
|
+
* generation settings are in range and sent by the target's provider adapter.
|
|
43
|
+
*/
|
|
44
|
+
export declare function validateExperimentFile(experimentFile: string): Promise<ExperimentValidation>;
|
|
45
|
+
/** Values accepted for a target's `settings.reasoning_effort`. */
|
|
46
|
+
export declare const REASONING_EFFORTS: readonly ["none", "minimal", "low", "medium", "high"];
|
|
47
|
+
export declare function isReasoningEffort(value: unknown): value is (typeof REASONING_EFFORTS)[number];
|
|
48
|
+
//# sourceMappingURL=experiment-validation.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"experiment-validation.d.ts","sourceRoot":"","sources":["../src/experiment-validation.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAIH,OAAO,EACL,KAAK,kBAAkB,EACvB,KAAK,cAAc,EAWpB,MAAM,kBAAkB,CAAC;AAI1B,MAAM,MAAM,sBAAsB,GAAG,MAAM,GAAG,UAAU,GAAG,MAAM,GAAG,QAAQ,CAAC;AAC7E,MAAM,MAAM,qBAAqB,GAAG,IAAI,GAAG,SAAS,GAAG,OAAO,CAAC;AAE/D,qEAAqE;AACrE,MAAM,WAAW,eAAe;IAC9B,OAAO,EAAE,sBAAsB,CAAC;IAChC,oDAAoD;IACpD,EAAE,CAAC,EAAE,MAAM,CAAC;IACZ,MAAM,EAAE,qBAAqB,CAAC;IAC9B,IAAI,EAAE,MAAM,CAAC;IACb,OAAO,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,oBAAoB;IACnC,0EAA0E;IAC1E,KAAK,EAAE,OAAO,CAAC;IACf,4CAA4C;IAC5C,IAAI,EAAE,MAAM,CAAC;IACb,wFAAwF;IACxF,IAAI,EAAE,MAAM,CAAC;IACb,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,IAAI,CAAC,EAAE,cAAc,CAAC;IACtB,sFAAsF;IACtF,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,2EAA2E;IAC3E,QAAQ,CAAC,EAAE,kBAAkB,CAAC;IAC9B,MAAM,EAAE,eAAe,EAAE,CAAC;IAC1B,2FAA2F;IAC3F,QAAQ,EAAE,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;CACrB;AAED,eAAO,MAAM,yBAAyB,gEACyB,CAAC;AAEhE;;;;;;;;GAQG;AACH,wBAAsB,sBAAsB,CAC1C,cAAc,EAAE,MAAM,GACrB,OAAO,CAAC,oBAAoB,CAAC,CA4D/B;AAwGD,kEAAkE;AAClE,eAAO,MAAM,iBAAiB,uDAAwD,CAAC;AAEvF,wBAAgB,iBAAiB,CAAC,KAAK,EAAE,OAAO,GAAG,KAAK,IAAI,CAAC,OAAO,iBAAiB,CAAC,CAAC,MAAM,CAAC,CAE7F"}
|
|
@@ -0,0 +1,162 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Comparative experiment surfaces: validate, run and report.
|
|
3
|
+
*
|
|
4
|
+
* This module is the one engine behind both the SDK functions and the `akit experiment` CLI.
|
|
5
|
+
* Fixture experiments bind every target to a deterministic in-memory client and make no network
|
|
6
|
+
* calls. Live experiments additionally require an explicit `ExperimentLiveAuthorization`.
|
|
7
|
+
*/
|
|
8
|
+
import { type AdapterConfig, type ExperimentLiveAuthorization, type ExperimentRunResult, type ExperimentTarget, type ModelClient } from '@artemiskit/core';
|
|
9
|
+
import { type ExperimentCaseLevelSummary, type ExperimentReportFormat, type ExperimentReportView } from '@artemiskit/reports';
|
|
10
|
+
import { type ExperimentExitCode, type ExperimentRunStatus } from './experiment-summary.js';
|
|
11
|
+
import { type ExperimentValidation } from './experiment-validation.js';
|
|
12
|
+
export { ExperimentSurfaceError } from './experiment-files.js';
|
|
13
|
+
export { EXPERIMENT_EXIT_CODES, describeConcurrency, describeTargetDeployment, experimentRunStatus, formatExperimentCaseSummary, formatExperimentRunSummary, } from './experiment-summary.js';
|
|
14
|
+
export type { ExperimentExitCode, ExperimentRunStatus } from './experiment-summary.js';
|
|
15
|
+
export { LOOPBACK_REQUIRED_MESSAGE, validateExperimentFile } from './experiment-validation.js';
|
|
16
|
+
export type { ExperimentCheck, ExperimentCheckStatus, ExperimentCheckSubject, ExperimentValidation, } from './experiment-validation.js';
|
|
17
|
+
export { createExperimentFixtureClient } from './experiment-fixture.js';
|
|
18
|
+
export type { ExperimentFixtureClientOptions } from './experiment-fixture.js';
|
|
19
|
+
/**
|
|
20
|
+
* Non-secret API key sent to loopback OpenAI-compatible servers (such as Ollama) that ignore
|
|
21
|
+
* authentication. It keeps the OpenAI client from falling back to an ambient `OPENAI_API_KEY`,
|
|
22
|
+
* so a real key never reaches a local endpoint unless the host configures one explicitly.
|
|
23
|
+
*/
|
|
24
|
+
export declare const LOCAL_ENDPOINT_PLACEHOLDER_API_KEY = "artemiskit-local-no-key";
|
|
25
|
+
export interface CreateExperimentLiveAuthorizationOptions {
|
|
26
|
+
/** Who approved the live run. */
|
|
27
|
+
approvedBy: string;
|
|
28
|
+
/** Why the live run is approved. */
|
|
29
|
+
reason: string;
|
|
30
|
+
/** Bounded decision ID; generated when omitted. */
|
|
31
|
+
decisionId?: string;
|
|
32
|
+
/** Decision time; defaults to now. */
|
|
33
|
+
decidedAt?: Date;
|
|
34
|
+
/**
|
|
35
|
+
* Highest experiment concurrency this approval covers (integer 1..8). Required to run a
|
|
36
|
+
* live experiment that declares concurrency above 1; omitted means serial live calls only.
|
|
37
|
+
*/
|
|
38
|
+
maxConcurrency?: number;
|
|
39
|
+
}
|
|
40
|
+
/**
|
|
41
|
+
* Build the explicit live-run approval record that `runExperimentFile` requires for
|
|
42
|
+
* `mode: live` experiments. Throws `ExperimentSurfaceError` (exit code 3) on invalid input.
|
|
43
|
+
*/
|
|
44
|
+
export declare function createExperimentLiveAuthorization(options: CreateExperimentLiveAuthorizationOptions): ExperimentLiveAuthorization;
|
|
45
|
+
export interface RunExperimentFileOptions {
|
|
46
|
+
/** Experiment YAML or JSON file. Task sources resolve relative to its directory. */
|
|
47
|
+
experiment: string;
|
|
48
|
+
/**
|
|
49
|
+
* New or empty directory for `result.json` and `evidence/`. Defaults to
|
|
50
|
+
* `<cwd>/artemis-output/experiments/<experiment-id>/<timestamp>/`.
|
|
51
|
+
*/
|
|
52
|
+
outputDir?: string;
|
|
53
|
+
/** Base for the default output directory; defaults to `process.cwd()`. */
|
|
54
|
+
cwd?: string;
|
|
55
|
+
/** Required for `mode: live`; ignored for fixture experiments. */
|
|
56
|
+
liveAuthorization?: ExperimentLiveAuthorization;
|
|
57
|
+
/**
|
|
58
|
+
* Live mode only: provider transport settings for a target (API key, timeout, organization).
|
|
59
|
+
* The target's `settings.base_url`, provider and model always take precedence.
|
|
60
|
+
*/
|
|
61
|
+
resolveTargetConfig?: (target: ExperimentTarget) => Partial<AdapterConfig> | undefined | Promise<Partial<AdapterConfig> | undefined>;
|
|
62
|
+
/** Live mode only: supply your own client per target instead of the built-in adapters. */
|
|
63
|
+
createClient?: (target: ExperimentTarget) => ModelClient | Promise<ModelClient>;
|
|
64
|
+
/**
|
|
65
|
+
* Per-case model request timeout in milliseconds for scenario tasks. Live targets bound
|
|
66
|
+
* through the built-in adapters also get an HTTP timeout of at least this value. Workflow
|
|
67
|
+
* tasks keep their own declared `timeout_ms`.
|
|
68
|
+
*/
|
|
69
|
+
timeoutMs?: number;
|
|
70
|
+
}
|
|
71
|
+
/**
|
|
72
|
+
* Saved evidence for one executed attempt: the scenario run manifest, or the metadata-only
|
|
73
|
+
* workflow record, as canonical JSON named by its SHA-256.
|
|
74
|
+
*/
|
|
75
|
+
export interface ExperimentEvidenceFile {
|
|
76
|
+
coordinate_id: string;
|
|
77
|
+
attempt_number: number;
|
|
78
|
+
sha256: string;
|
|
79
|
+
path: string;
|
|
80
|
+
}
|
|
81
|
+
export interface ExperimentFileRunOutcome {
|
|
82
|
+
status: ExperimentRunStatus;
|
|
83
|
+
/** Same value `akit experiment run` exits with. */
|
|
84
|
+
exitCode: ExperimentExitCode;
|
|
85
|
+
validation: ExperimentValidation;
|
|
86
|
+
/** Present once the run executed. */
|
|
87
|
+
result?: ExperimentRunResult;
|
|
88
|
+
outputDir?: string;
|
|
89
|
+
resultPath?: string;
|
|
90
|
+
evidence: ExperimentEvidenceFile[];
|
|
91
|
+
/**
|
|
92
|
+
* Present once the run executed: individual scenario cases from the saved, SHA-256-verified
|
|
93
|
+
* evidence, or why they are unavailable. Coordinate-level counts stay in `result.summaries`.
|
|
94
|
+
*/
|
|
95
|
+
caseLevel?: ExperimentCaseLevelSummary;
|
|
96
|
+
/** Explanation for refusals (`invalid`, `unauthorized`). */
|
|
97
|
+
message?: string;
|
|
98
|
+
}
|
|
99
|
+
/**
|
|
100
|
+
* Validate, then run an experiment file and save the V1 result plus per-attempt evidence.
|
|
101
|
+
*
|
|
102
|
+
* Each task runs through the built-in executor for its kind: `scenario_evaluation` tasks
|
|
103
|
+
* through the scenario runner and `agent_workflow` tasks through the native workflow engine
|
|
104
|
+
* (simulated or sandbox environment, network denied). Fixture targets never call a tool, so
|
|
105
|
+
* fixture workflow outcomes describe the pipeline, never a model.
|
|
106
|
+
*
|
|
107
|
+
* Refusals (`invalid`, `unauthorized`) return before any client is created or any attempt runs.
|
|
108
|
+
* Output and persistence failures throw `ExperimentSurfaceError` (exit code 1).
|
|
109
|
+
*
|
|
110
|
+
* A target's generation settings (`temperature`, `top_p`, `presence_penalty`,
|
|
111
|
+
* `max_output_tokens`) replace each task's own values for that target's requests, and every
|
|
112
|
+
* request carries the coordinate's seed. `llm_grader` and embedding requests keep their own
|
|
113
|
+
* sampling and output length. `reasoning_effort` is adapter configuration, so it also applies to
|
|
114
|
+
* `llm_grader` requests made through the target's client. Workflow tasks get the same settings
|
|
115
|
+
* and seed on every workflow model request. Each saved run manifest records the values sent in
|
|
116
|
+
* `resolved_config`; each saved workflow record lists them in `configuration.generation`.
|
|
117
|
+
*/
|
|
118
|
+
export declare function runExperimentFile(options: RunExperimentFileOptions): Promise<ExperimentFileRunOutcome>;
|
|
119
|
+
export interface ReportExperimentResultOptions {
|
|
120
|
+
/** Path to a saved `result.json`, or the parsed V1 result. */
|
|
121
|
+
result: string | ExperimentRunResult;
|
|
122
|
+
/** Views to generate; defaults to all three. */
|
|
123
|
+
views?: ExperimentReportView[];
|
|
124
|
+
/** Formats to generate; defaults to HTML and Markdown. */
|
|
125
|
+
formats?: ExperimentReportFormat[];
|
|
126
|
+
/** New or empty directory to write `<view>.<html|md>` files into; omit to only return content. */
|
|
127
|
+
outputDir?: string;
|
|
128
|
+
/**
|
|
129
|
+
* Directory of saved per-attempt evidence for the case-level summary. Defaults to `evidence/`
|
|
130
|
+
* beside a result file; a parsed result without this option reports case level unavailable.
|
|
131
|
+
*/
|
|
132
|
+
evidenceDir?: string;
|
|
133
|
+
}
|
|
134
|
+
export interface ExperimentReportFile {
|
|
135
|
+
view: ExperimentReportView;
|
|
136
|
+
format: ExperimentReportFormat;
|
|
137
|
+
content: string;
|
|
138
|
+
/** Written file, when `outputDir` was given. */
|
|
139
|
+
path?: string;
|
|
140
|
+
}
|
|
141
|
+
export declare const EXPERIMENT_REPORT_VIEWS: readonly ExperimentReportView[];
|
|
142
|
+
export declare const EXPERIMENT_REPORT_FORMATS: readonly ExperimentReportFormat[];
|
|
143
|
+
/**
|
|
144
|
+
* Generate experiment reports offline from a saved result. Loads no task, provider, credential
|
|
145
|
+
* or network. Throws `ExperimentSurfaceError`: exit code 1 for unreadable input or unwritable
|
|
146
|
+
* output, 2 for an invalid or unsupported saved result.
|
|
147
|
+
*/
|
|
148
|
+
export declare function reportExperimentResult(options: ReportExperimentResultOptions): Promise<ExperimentReportFile[]>;
|
|
149
|
+
export interface SummarizeExperimentCasesOptions {
|
|
150
|
+
/** Path to a saved `result.json`, or the parsed V1 result. */
|
|
151
|
+
result: string | ExperimentRunResult;
|
|
152
|
+
/** Saved per-attempt evidence directory; defaults to `evidence/` beside a result file. */
|
|
153
|
+
evidenceDir?: string;
|
|
154
|
+
}
|
|
155
|
+
/**
|
|
156
|
+
* Case-level summary of a saved result, offline: individual scenario cases from each
|
|
157
|
+
* coordinate's terminal attempt evidence, verified against the recorded SHA-256. Missing or
|
|
158
|
+
* altered evidence yields an `unavailable` summary with its reason; an invalid result throws
|
|
159
|
+
* `ExperimentSurfaceError` (exit code 1 unreadable, 2 invalid).
|
|
160
|
+
*/
|
|
161
|
+
export declare function summarizeExperimentCases(options: SummarizeExperimentCasesOptions): Promise<ExperimentCaseLevelSummary>;
|
|
162
|
+
//# sourceMappingURL=experiment.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"experiment.d.ts","sourceRoot":"","sources":["../src/experiment.ts"],"names":[],"mappings":"AAAA;;;;;;GAMG;AAIH,OAAO,EACL,KAAK,aAAa,EAGlB,KAAK,2BAA2B,EAChC,KAAK,mBAAmB,EACxB,KAAK,gBAAgB,EAErB,KAAK,WAAW,EAMjB,MAAM,kBAAkB,CAAC;AAC1B,OAAO,EAEL,KAAK,0BAA0B,EAC/B,KAAK,sBAAsB,EAC3B,KAAK,oBAAoB,EAM1B,MAAM,qBAAqB,CAAC;AAc7B,OAAO,EAEL,KAAK,kBAAkB,EACvB,KAAK,mBAAmB,EAEzB,MAAM,sBAAsB,CAAC;AAC9B,OAAO,EACL,KAAK,oBAAoB,EAG1B,MAAM,yBAAyB,CAAC;AAEjC,OAAO,EAAE,sBAAsB,EAAE,MAAM,oBAAoB,CAAC;AAC5D,OAAO,EACL,qBAAqB,EACrB,mBAAmB,EACnB,wBAAwB,EACxB,mBAAmB,EACnB,2BAA2B,EAC3B,0BAA0B,GAC3B,MAAM,sBAAsB,CAAC;AAC9B,YAAY,EAAE,kBAAkB,EAAE,mBAAmB,EAAE,MAAM,sBAAsB,CAAC;AACpF,OAAO,EAAE,yBAAyB,EAAE,sBAAsB,EAAE,MAAM,yBAAyB,CAAC;AAC5F,YAAY,EACV,eAAe,EACf,qBAAqB,EACrB,sBAAsB,EACtB,oBAAoB,GACrB,MAAM,yBAAyB,CAAC;AACjC,OAAO,EAAE,6BAA6B,EAAE,MAAM,sBAAsB,CAAC;AACrE,YAAY,EAAE,8BAA8B,EAAE,MAAM,sBAAsB,CAAC;AAE3E;;;;GAIG;AACH,eAAO,MAAM,kCAAkC,4BAA4B,CAAC;AAQ5E,MAAM,WAAW,wCAAwC;IACvD,iCAAiC;IACjC,UAAU,EAAE,MAAM,CAAC;IACnB,oCAAoC;IACpC,MAAM,EAAE,MAAM,CAAC;IACf,mDAAmD;IACnD,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,sCAAsC;IACtC,SAAS,CAAC,EAAE,IAAI,CAAC;IACjB;;;OAGG;IACH,cAAc,CAAC,EAAE,MAAM,CAAC;CACzB;AAED;;;GAGG;AACH,wBAAgB,iCAAiC,CAC/C,OAAO,EAAE,wCAAwC,GAChD,2BAA2B,CAoB7B;AA8BD,MAAM,WAAW,wBAAwB;IACvC,oFAAoF;IACpF,UAAU,EAAE,MAAM,CAAC;IACnB;;;OAGG;IACH,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,0EAA0E;IAC1E,GAAG,CAAC,EAAE,MAAM,CAAC;IACb,kEAAkE;IAClE,iBAAiB,CAAC,EAAE,2BAA2B,CAAC;IAChD;;;OAGG;IACH,mBAAmB,CAAC,EAAE,CACpB,MAAM,EAAE,gBAAgB,KACrB,OAAO,CAAC,aAAa,CAAC,GAAG,SAAS,GAAG,OAAO,CAAC,OAAO,CAAC,aAAa,CAAC,GAAG,SAAS,CAAC,CAAC;IACtF,0FAA0F;IAC1F,YAAY,CAAC,EAAE,CAAC,MAAM,EAAE,gBAAgB,KAAK,WAAW,GAAG,OAAO,CAAC,WAAW,CAAC,CAAC;IAChF;;;;OAIG;IACH,SAAS,CAAC,EAAE,MAAM,CAAC;CACpB;AAED;;;GAGG;AACH,MAAM,WAAW,sBAAsB;IACrC,aAAa,EAAE,MAAM,CAAC;IACtB,cAAc,EAAE,MAAM,CAAC;IACvB,MAAM,EAAE,MAAM,CAAC;IACf,IAAI,EAAE,MAAM,CAAC;CACd;AAED,MAAM,WAAW,wBAAwB;IACvC,MAAM,EAAE,mBAAmB,CAAC;IAC5B,mDAAmD;IACnD,QAAQ,EAAE,kBAAkB,CAAC;IAC7B,UAAU,EAAE,oBAAoB,CAAC;IACjC,qCAAqC;IACrC,MAAM,CAAC,EAAE,mBAAmB,CAAC;IAC7B,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,QAAQ,EAAE,sBAAsB,EAAE,CAAC;IACnC;;;OAGG;IACH,SAAS,CAAC,EAAE,0BAA0B,CAAC;IACvC,4DAA4D;IAC5D,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED;;;;;;;;;;;;;;;;;;GAkBG;AACH,wBAAsB,iBAAiB,CACrC,OAAO,EAAE,wBAAwB,GAChC,OAAO,CAAC,wBAAwB,CAAC,CA8EnC;AAwJD,MAAM,WAAW,6BAA6B;IAC5C,8DAA8D;IAC9D,MAAM,EAAE,MAAM,GAAG,mBAAmB,CAAC;IACrC,gDAAgD;IAChD,KAAK,CAAC,EAAE,oBAAoB,EAAE,CAAC;IAC/B,0DAA0D;IAC1D,OAAO,CAAC,EAAE,sBAAsB,EAAE,CAAC;IACnC,kGAAkG;IAClG,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB;;;OAGG;IACH,WAAW,CAAC,EAAE,MAAM,CAAC;CACtB;AAED,MAAM,WAAW,oBAAoB;IACnC,IAAI,EAAE,oBAAoB,CAAC;IAC3B,MAAM,EAAE,sBAAsB,CAAC;IAC/B,OAAO,EAAE,MAAM,CAAC;IAChB,gDAAgD;IAChD,IAAI,CAAC,EAAE,MAAM,CAAC;CACf;AAED,eAAO,MAAM,uBAAuB,EAAE,SAAS,oBAAoB,EAIlE,CAAC;AACF,eAAO,MAAM,yBAAyB,EAAE,SAAS,sBAAsB,EAAyB,CAAC;AAEjG;;;;GAIG;AACH,wBAAsB,sBAAsB,CAC1C,OAAO,EAAE,6BAA6B,GACrC,OAAO,CAAC,oBAAoB,EAAE,CAAC,CAgDjC;AAED,MAAM,WAAW,+BAA+B;IAC9C,8DAA8D;IAC9D,MAAM,EAAE,MAAM,GAAG,mBAAmB,CAAC;IACrC,0FAA0F;IAC1F,WAAW,CAAC,EAAE,MAAM,CAAC;CACtB;AAED;;;;;GAKG;AACH,wBAAsB,wBAAwB,CAC5C,OAAO,EAAE,+BAA+B,GACvC,OAAO,CAAC,0BAA0B,CAAC,CAgBrC"}
|
|
@@ -26,7 +26,12 @@ export declare class CircuitBreaker {
|
|
|
26
26
|
private lastOpenTime;
|
|
27
27
|
private halfOpenSuccesses;
|
|
28
28
|
private eventHandlers;
|
|
29
|
-
|
|
29
|
+
/**
|
|
30
|
+
* @param config Circuit breaker settings. Every field is optional; omitted fields default to
|
|
31
|
+
* `enabled: true`, `threshold: 5`, `windowMs: 60000`, `cooldownMs: 300000` and
|
|
32
|
+
* `halfOpenRequests: 3`.
|
|
33
|
+
*/
|
|
34
|
+
constructor(config: Partial<CircuitBreakerConfig>);
|
|
30
35
|
/**
|
|
31
36
|
* Check if the circuit is open (requests should be blocked)
|
|
32
37
|
*/
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"circuit-breaker.d.ts","sourceRoot":"","sources":["../../src/guardian/circuit-breaker.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AAEH,OAAO,KAAK,EACV,oBAAoB,EACpB,mBAAmB,EAEnB,eAAe,EAEf,SAAS,EAEV,MAAM,SAAS,CAAC;AAMjB;;GAEG;AACH,MAAM,MAAM,mBAAmB,GAAG,MAAM,GAAG,OAAO,GAAG,WAAW,GAAG,WAAW,GAAG,SAAS,CAAC;AAE3F;;GAEG;AACH,MAAM,MAAM,0BAA0B,GAAG,CACvC,KAAK,EAAE,mBAAmB,EAC1B,IAAI,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,KAC3B,IAAI,CAAC;AAEV;;;;;GAKG;AACH,qBAAa,cAAc;IACzB,OAAO,CAAC,MAAM,CAAiC;IAC/C,OAAO,CAAC,KAAK,CAAiC;IAC9C,OAAO,CAAC,UAAU,CAA4D;IAC9E,OAAO,CAAC,YAAY,CAAK;IACzB,OAAO,CAAC,iBAAiB,CAAK;IAC9B,OAAO,CAAC,aAAa,CAAoC;
|
|
1
|
+
{"version":3,"file":"circuit-breaker.d.ts","sourceRoot":"","sources":["../../src/guardian/circuit-breaker.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AAEH,OAAO,KAAK,EACV,oBAAoB,EACpB,mBAAmB,EAEnB,eAAe,EAEf,SAAS,EAEV,MAAM,SAAS,CAAC;AAMjB;;GAEG;AACH,MAAM,MAAM,mBAAmB,GAAG,MAAM,GAAG,OAAO,GAAG,WAAW,GAAG,WAAW,GAAG,SAAS,CAAC;AAE3F;;GAEG;AACH,MAAM,MAAM,0BAA0B,GAAG,CACvC,KAAK,EAAE,mBAAmB,EAC1B,IAAI,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,KAC3B,IAAI,CAAC;AAEV;;;;;GAKG;AACH,qBAAa,cAAc;IACzB,OAAO,CAAC,MAAM,CAAiC;IAC/C,OAAO,CAAC,KAAK,CAAiC;IAC9C,OAAO,CAAC,UAAU,CAA4D;IAC9E,OAAO,CAAC,YAAY,CAAK;IACzB,OAAO,CAAC,iBAAiB,CAAK;IAC9B,OAAO,CAAC,aAAa,CAAoC;IAEzD;;;;OAIG;gBACS,MAAM,EAAE,OAAO,CAAC,oBAAoB,CAAC;IAUjD;;OAEG;IACH,MAAM,IAAI,OAAO;IAgBjB;;;;OAIG;IACH,YAAY,IAAI,OAAO;IAIvB;;OAEG;IACH,eAAe,CAAC,SAAS,EAAE,SAAS,GAAG,IAAI;IA6B3C;;OAEG;IACH,aAAa,IAAI,IAAI;IAgBrB;;OAEG;IACH,IAAI,IAAI,IAAI;IAKZ;;OAEG;IACH,KAAK,IAAI,IAAI;IAMb;;;;OAIG;IACH,UAAU,CAAC,KAAK,EAAE,mBAAmB,GAAG,IAAI;IAO5C;;OAEG;IACH,QAAQ,IAAI,mBAAmB;IAI/B;;OAEG;IACH,iBAAiB,IAAI,MAAM;IAK3B;;OAEG;IACH,iBAAiB,IAAI,MAAM;IAQ3B;;OAEG;IACH,OAAO,CAAC,OAAO,EAAE,0BAA0B,GAAG,IAAI;IAIlD;;OAEG;IACH,QAAQ,CAAC,OAAO,EAAE,0BAA0B,GAAG,IAAI;IAInD;;OAEG;IACH,OAAO,CAAC,UAAU;IAqBlB;;OAEG;IACH,OAAO,CAAC,IAAI;CASb;AAoBD;;GAEG;AACH,qBAAa,gBAAgB;IAC3B,OAAO,CAAC,aAAa,CAAgB;IACrC,OAAO,CAAC,gBAAgB,CAAS;IACjC,OAAO,CAAC,OAAO,CAAuB;IACtC,OAAO,CAAC,iBAAiB,CAAS;IAClC,OAAO,CAAC,QAAQ,CAAS;gBAGvB,OAAO,GAAE;QACP,gBAAgB,CAAC,EAAE,MAAM,CAAC;QAC1B,iBAAiB,CAAC,EAAE,MAAM,CAAC;QAC3B,QAAQ,CAAC,EAAE,MAAM,CAAC;KACd;IAQR;;OAEG;IACH,aAAa,CAAC,OAAO,EAAE;QACrB,OAAO,EAAE,OAAO,CAAC;QACjB,MAAM,EAAE,OAAO,CAAC;QAChB,SAAS,EAAE,MAAM,CAAC;QAClB,UAAU,EAAE,SAAS,EAAE,CAAC;QACxB,IAAI,CAAC,EAAE,MAAM,CAAC;KACf,GAAG,IAAI;IA8BR;;OAEG;IACH,UAAU,CAAC,mBAAmB,GAAE,mBAA8B,GAAG,eAAe;IAkEhF;;OAEG;IACH,uBAAuB,IAAI,OAAO,CAAC,eAAe,CAAC;IAanD;;OAEG;IACH,KAAK,IAAI,IAAI;IAKb;;;;;;OAMG;IACH,OAAO,CAAC,gBAAgB;IAyBxB;;OAEG;IACH,OAAO,CAAC,aAAa;IAMrB;;OAEG;IACH,OAAO,CAAC,YAAY;IAWpB;;OAEG;IACH,OAAO,CAAC,YAAY;IAapB;;OAEG;IACH,OAAO,CAAC,oBAAoB;CAa7B;AAMD;;GAEG;AACH,MAAM,WAAW,iBAAiB;IAChC,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAC3B,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,UAAU,CAAC,EAAE,MAAM,CAAC;CACrB;AAED;;GAEG;AACH,qBAAa,WAAW;IACtB,OAAO,CAAC,MAAM,CAAoB;IAClC,OAAO,CAAC,YAAY,CAAuC;IAC3D,OAAO,CAAC,UAAU,CAAuC;IACzD,OAAO,CAAC,SAAS,CAAuC;IACxD,OAAO,CAAC,WAAW,CAAS;IAC5B,OAAO,CAAC,eAAe,CAAS;gBAEpB,MAAM,EAAE,iBAAiB;IAUrC;;OAEG;IACH,YAAY,IAAI;QAAE,OAAO,EAAE,OAAO,CAAC;QAAC,MAAM,CAAC,EAAE,MAAM,CAAC;QAAC,YAAY,CAAC,EAAE,MAAM,CAAA;KAAE;IAgE5E;;OAEG;IACH,SAAS,IAAI;QACX,UAAU,EAAE,MAAM,CAAC;QACnB,WAAW,CAAC,EAAE,MAAM,CAAC;QACrB,QAAQ,EAAE,MAAM,CAAC;QACjB,SAAS,CAAC,EAAE,MAAM,CAAC;QACnB,OAAO,EAAE,MAAM,CAAC;QAChB,QAAQ,CAAC,EAAE,MAAM,CAAC;QAClB,WAAW,EAAE,MAAM,CAAC;QACpB,UAAU,CAAC,EAAE,MAAM,CAAC;KACrB;IAaD;;OAEG;IACH,KAAK,IAAI,IAAI;IASb;;OAEG;IACH,OAAO,CAAC,iBAAiB;IAezB;;;;;;;OAOG;IACH,OAAO,CAAC,aAAa;CA6BtB"}
|