evalsutra 0.0.0-stage → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/evalsutra-arena/SKILL.md +27 -0
- package/.agents/skills/evalsutra-audit/SKILL.md +27 -0
- package/.agents/skills/evalsutra-bench/SKILL.md +27 -0
- package/.agents/skills/evalsutra-cost/SKILL.md +27 -0
- package/.agents/skills/evalsutra-data/SKILL.md +27 -0
- package/.agents/skills/evalsutra-drift/SKILL.md +27 -0
- package/.agents/skills/evalsutra-explain/SKILL.md +27 -0
- package/.agents/skills/evalsutra-guard/SKILL.md +27 -0
- package/.agents/skills/evalsutra-judge/SKILL.md +27 -0
- package/.agents/skills/evalsutra-pluginwatch/SKILL.md +27 -0
- package/.agents/skills/evalsutra-port/SKILL.md +27 -0
- package/.agents/skills/evalsutra-prime/SKILL.md +27 -0
- package/.agents/skills/evalsutra-report/SKILL.md +27 -0
- package/.agents/skills/evalsutra-robust/SKILL.md +27 -0
- package/.agents/skills/evalsutra-task/SKILL.md +27 -0
- package/.agents/skills/evalsutra-trace/SKILL.md +27 -0
- package/.agents/skills/evalsutra-truth/SKILL.md +27 -0
- package/.agents/skills/evalsutra-verify/SKILL.md +27 -0
- package/.claude-plugin/plugin.json +8 -0
- package/CHANGELOG.md +33 -0
- package/CITATION.cff +14 -0
- package/CODE_OF_CONDUCT.md +2 -0
- package/CONTRIBUTING.md +2 -0
- package/LICENSE +202 -0
- package/NOTICE +5 -0
- package/README.md +161 -2
- package/SECURITY.md +17 -0
- package/SUPPORT.md +5 -0
- package/THIRD_PARTY_NOTICES.md +14 -0
- package/docs/ADVANCED_TOOLING_ROADMAP.md +244 -0
- package/docs/ARCHITECTURE.md +17 -0
- package/docs/PLUGIN_AND_SKILL_CATALOG.md +56 -0
- package/docs/STATUS.md +25 -0
- package/docs/master-claude-prompt.md +534 -0
- package/documents/evalsutra/01-executive-summary.md +52 -0
- package/documents/evalsutra/02-functional-specification.md +104 -0
- package/documents/evalsutra/03-architecture.md +78 -0
- package/documents/evalsutra/04-technical-reference.md +150 -0
- package/documents/evalsutra/05-user-guide.md +94 -0
- package/documents/evalsutra/06-skills-catalog.md +52 -0
- package/documents/evalsutra/07-security-and-legal.md +66 -0
- package/documents/evalsutra/08-testing-and-evidence.md +63 -0
- package/documents/evalsutra/09-audit-and-trust.md +54 -0
- package/documents/evalsutra/10-ci-cd-and-release.md +50 -0
- package/documents/evalsutra/11-faq-and-glossary.md +51 -0
- package/documents/evalsutra/12-contributor-handbook.md +67 -0
- package/documents/evalsutra/13-roadmap-and-status.md +48 -0
- package/documents/evalsutra/assets/architecture.svg +21 -0
- package/documents/evalsutra/evidence/README.md +39 -0
- package/documents/evalsutra/evidence/samples/audit/audit-report.html +46 -0
- package/documents/evalsutra/evidence/samples/audit/audit-report.json +317 -0
- package/documents/evalsutra/evidence/samples/audit/audit-report.md +64 -0
- package/documents/evalsutra/evidence/samples/bad/junit.xml +2 -0
- package/documents/evalsutra/evidence/samples/bad/report.html +47 -0
- package/documents/evalsutra/evidence/samples/bad/report.json +153 -0
- package/documents/evalsutra/evidence/samples/bad/report.md +22 -0
- package/documents/evalsutra/evidence/samples/compare-good-vs-bad.json +59 -0
- package/documents/evalsutra/evidence/samples/demo-suite.json +13 -0
- package/documents/evalsutra/evidence/samples/gate-bad-run.json +10 -0
- package/documents/evalsutra/evidence/samples/good/junit.xml +2 -0
- package/documents/evalsutra/evidence/samples/good/report.html +47 -0
- package/documents/evalsutra/evidence/samples/good/report.json +153 -0
- package/documents/evalsutra/evidence/samples/good/report.md +22 -0
- package/documents/evalsutra/evidence/samples/trace-analysis.json +33 -0
- package/documents/evalsutra/evidence/test-run.txt +71 -0
- package/documents/evalsutra/evidence/test-summary.json +11 -0
- package/documents/evalsutra/html/01-executive-summary.html +57 -0
- package/documents/evalsutra/html/02-functional-specification.html +75 -0
- package/documents/evalsutra/html/03-architecture.html +60 -0
- package/documents/evalsutra/html/04-technical-reference.html +109 -0
- package/documents/evalsutra/html/05-user-guide.html +72 -0
- package/documents/evalsutra/html/06-skills-catalog.html +54 -0
- package/documents/evalsutra/html/07-security-and-legal.html +63 -0
- package/documents/evalsutra/html/08-testing-and-evidence.html +61 -0
- package/documents/evalsutra/html/09-audit-and-trust.html +61 -0
- package/documents/evalsutra/html/10-ci-cd-and-release.html +61 -0
- package/documents/evalsutra/html/11-faq-and-glossary.html +57 -0
- package/documents/evalsutra/html/12-contributor-handbook.html +71 -0
- package/documents/evalsutra/html/13-roadmap-and-status.html +51 -0
- package/documents/evalsutra/html/assets/architecture.svg +21 -0
- package/documents/evalsutra/html/index.html +42 -0
- package/examples/demo-observations-bad.json +7 -0
- package/examples/demo-observations.json +7 -0
- package/examples/observations.json +10 -0
- package/examples/output/report.json +99 -0
- package/examples/output/report.md +18 -0
- package/examples/suite.json +11 -0
- package/examples/traces/bad-trace.json +5 -0
- package/llms.txt +25 -0
- package/package.json +19 -4
- package/packages/audit/audit.mjs +179 -0
- package/packages/cli/evalsutra.mjs +151 -0
- package/packages/cli/install.mjs +68 -0
- package/packages/core/capsule.mjs +31 -0
- package/packages/core/evaluate.mjs +187 -0
- package/packages/core/security.mjs +22 -0
- package/packages/core/stats.mjs +16 -0
- package/packages/core/trace.mjs +29 -0
- package/packages/reporters/index.mjs +26 -0
- package/packages/reporters/theme.mjs +53 -0
- package/policies/README.md +3 -0
- package/policies/default.json +1 -0
- package/skills/evalsutra-arena/SKILL.md +27 -0
- package/skills/evalsutra-audit/SKILL.md +27 -0
- package/skills/evalsutra-bench/SKILL.md +27 -0
- package/skills/evalsutra-cost/SKILL.md +27 -0
- package/skills/evalsutra-data/SKILL.md +27 -0
- package/skills/evalsutra-drift/SKILL.md +27 -0
- package/skills/evalsutra-explain/SKILL.md +27 -0
- package/skills/evalsutra-guard/SKILL.md +27 -0
- package/skills/evalsutra-judge/SKILL.md +27 -0
- package/skills/evalsutra-pluginwatch/SKILL.md +27 -0
- package/skills/evalsutra-port/SKILL.md +27 -0
- package/skills/evalsutra-prime/SKILL.md +27 -0
- package/skills/evalsutra-report/SKILL.md +27 -0
- package/skills/evalsutra-robust/SKILL.md +27 -0
- package/skills/evalsutra-task/SKILL.md +27 -0
- package/skills/evalsutra-trace/SKILL.md +27 -0
- package/skills/evalsutra-truth/SKILL.md +27 -0
- package/skills/evalsutra-verify/SKILL.md +27 -0
- package/suites/demo.json +13 -0
|
@@ -0,0 +1,187 @@
|
|
|
1
|
+
import { readFileSync } from 'node:fs';
|
|
2
|
+
import { wilson } from './stats.mjs';
|
|
3
|
+
|
|
4
|
+
export const SCORING_VERSION = 'scoring-v2';
|
|
5
|
+
export const SOURCE_TYPES = ['actual_run', 'imported_trace', 'human_label', 'test_fixture', 'model_judge', 'unverified_claim'];
|
|
6
|
+
// Sources that are opinions or claims: they may reveal failures but never prove a pass.
|
|
7
|
+
const WEAK_SOURCES = new Set(['model_judge', 'unverified_claim']);
|
|
8
|
+
const TEXT = ['equals', 'normalized_equals', 'contains', 'not_contains', 'regex', 'json_has_keys'];
|
|
9
|
+
const TOOLS = ['required_tools', 'forbidden_tools', 'tool_order'];
|
|
10
|
+
export const ASSERTION_TYPES = [...TEXT, ...TOOLS, 'max_steps', 'max_latency_ms'];
|
|
11
|
+
|
|
12
|
+
export function readJson(path) {
|
|
13
|
+
try { return JSON.parse(readFileSync(path, 'utf8')); }
|
|
14
|
+
catch (e) { throw new Error(`cannot read JSON ${path}: ${e.message}`); }
|
|
15
|
+
}
|
|
16
|
+
|
|
17
|
+
// shortcut: heuristic ReDoS guard (length cap + nested quantifier/backreference ban); upgrade to RE2 if untrusted suites become common.
|
|
18
|
+
export function unsafeRegex(p) {
|
|
19
|
+
return typeof p !== 'string' || !p || p.length > 200 || /\([^)]*[+*][^)]*\)[+*{]/.test(p) || /\\[1-9]/.test(p);
|
|
20
|
+
}
|
|
21
|
+
|
|
22
|
+
export function validateSuite(suite) {
|
|
23
|
+
const errors = [];
|
|
24
|
+
if (suite?.schemaVersion !== 1) errors.push('schemaVersion must equal 1');
|
|
25
|
+
if (typeof suite?.name !== 'string' || !suite.name.trim()) errors.push('suite.name is required');
|
|
26
|
+
if (!Array.isArray(suite?.cases) || suite.cases.length === 0) errors.push('cases must be a nonempty array');
|
|
27
|
+
const ids = new Set();
|
|
28
|
+
for (const item of (Array.isArray(suite?.cases) ? suite.cases : [])) {
|
|
29
|
+
if (!item?.id || typeof item.id !== 'string' || !/^[a-z0-9][a-z0-9-]*$/.test(item.id)) errors.push('case.id must be kebab-case');
|
|
30
|
+
if (ids.has(item.id)) errors.push(`duplicate case ID: ${item.id}`);
|
|
31
|
+
ids.add(item.id);
|
|
32
|
+
if (!Array.isArray(item.assertions) || !item.assertions.length) errors.push(`case ${item.id}: assertions required`);
|
|
33
|
+
for (const a of (Array.isArray(item.assertions) ? item.assertions : [])) {
|
|
34
|
+
if (!ASSERTION_TYPES.includes(a?.type)) { errors.push(`case ${item.id}: unknown assertion type ${a?.type}`); continue; }
|
|
35
|
+
if (a.weight !== undefined && (!Number.isFinite(a.weight) || a.weight <= 0)) errors.push(`case ${item.id}: assertion weight must be >0`);
|
|
36
|
+
if (['equals', 'normalized_equals', 'contains', 'not_contains'].includes(a.type) && typeof a.value !== 'string') errors.push(`case ${item.id}: ${a.type} needs string value`);
|
|
37
|
+
if (a.type === 'regex' && unsafeRegex(a.value)) errors.push(`case ${item.id}: regex missing, too long (>200) or unsafe`);
|
|
38
|
+
if (a.type === 'json_has_keys' && (!Array.isArray(a.keys) || !a.keys.length || !a.keys.every(k => typeof k === 'string'))) errors.push(`case ${item.id}: json_has_keys needs string keys array`);
|
|
39
|
+
if (TOOLS.includes(a.type) && (!Array.isArray(a.tools) || !a.tools.every(t => typeof t === 'string'))) errors.push(`case ${item.id}: ${a.type} needs string tools array`);
|
|
40
|
+
if (['max_steps', 'max_latency_ms'].includes(a.type) && (!Number.isFinite(a.value) || a.value < 0)) errors.push(`case ${item.id}: ${a.type} needs numeric value >=0`);
|
|
41
|
+
}
|
|
42
|
+
}
|
|
43
|
+
const t = suite?.gate || {};
|
|
44
|
+
for (const key of ['minPassRate', 'minCoverage']) if (t[key] !== undefined && (!Number.isFinite(t[key]) || t[key] < 0 || t[key] > 1)) errors.push(`gate.${key} must be between 0 and 1`);
|
|
45
|
+
return errors;
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
const norm = s => s.normalize('NFKC').trim().replace(/\s+/g, ' ').toLowerCase();
|
|
49
|
+
const has = (obj, field) => obj && Object.hasOwn(obj, field);
|
|
50
|
+
|
|
51
|
+
export function evaluateAssertion(assertion, observation) {
|
|
52
|
+
const mk = (status, reason, detail) => ({ status, reason, detail });
|
|
53
|
+
const unknown = (reason, detail) => mk('inconclusive', reason, detail);
|
|
54
|
+
const ok = (pass, detail) => mk(pass ? 'pass' : 'fail', pass ? 'OK' : 'MISMATCH', detail);
|
|
55
|
+
if (!observation || typeof observation !== 'object') return unknown('OBSERVATION_MISSING', 'No recorded observation');
|
|
56
|
+
if (TEXT.includes(assertion.type)) {
|
|
57
|
+
if (!has(observation, 'output') || typeof observation.output !== 'string') return unknown('OUTPUT_MISSING', 'output missing');
|
|
58
|
+
const actual = observation.output;
|
|
59
|
+
switch (assertion.type) {
|
|
60
|
+
case 'equals': return ok(actual === assertion.value, 'Exact answer comparison');
|
|
61
|
+
case 'normalized_equals': return ok(norm(actual) === norm(assertion.value), 'Normalized answer comparison');
|
|
62
|
+
case 'contains': return ok(actual.includes(assertion.value), 'Required text comparison');
|
|
63
|
+
case 'not_contains': return ok(!actual.includes(assertion.value), 'Forbidden text comparison');
|
|
64
|
+
case 'regex':
|
|
65
|
+
if (unsafeRegex(assertion.value)) return mk('error', 'UNSAFE_REGEX', 'Regex rejected by safety constraints');
|
|
66
|
+
try { return ok(new RegExp(assertion.value, 'u').test(actual.slice(0, 100000)), 'Regex match'); }
|
|
67
|
+
catch { return mk('error', 'BAD_REGEX', 'Regex failed to compile'); }
|
|
68
|
+
default: { // json_has_keys
|
|
69
|
+
let parsed;
|
|
70
|
+
try { parsed = JSON.parse(actual); } catch { return ok(false, 'Output is not valid JSON'); }
|
|
71
|
+
const isObj = parsed && typeof parsed === 'object' && !Array.isArray(parsed);
|
|
72
|
+
return ok(isObj && assertion.keys.every(k => Object.hasOwn(parsed, k)), 'JSON required keys');
|
|
73
|
+
}
|
|
74
|
+
}
|
|
75
|
+
}
|
|
76
|
+
if (TOOLS.includes(assertion.type) || assertion.type === 'max_steps') {
|
|
77
|
+
if (!Array.isArray(observation.toolCalls) || !observation.toolCalls.every(x => typeof x === 'string')) return unknown('TRACE_MISSING', 'toolCalls missing');
|
|
78
|
+
const actual = observation.toolCalls;
|
|
79
|
+
if (assertion.type === 'required_tools') return ok(assertion.tools.every(t => actual.includes(t)), 'Required tool use');
|
|
80
|
+
if (assertion.type === 'forbidden_tools') return ok(assertion.tools.every(t => !actual.includes(t)), 'Forbidden tool use');
|
|
81
|
+
if (assertion.type === 'max_steps') return ok(actual.length <= assertion.value, 'Maximum tool steps');
|
|
82
|
+
let cursor = -1;
|
|
83
|
+
for (const tool of assertion.tools) { cursor = actual.indexOf(tool, cursor + 1); if (cursor < 0) return ok(false, 'Expected tool order'); }
|
|
84
|
+
return ok(true, 'Expected tool order');
|
|
85
|
+
}
|
|
86
|
+
if (assertion.type === 'max_latency_ms') {
|
|
87
|
+
if (!Number.isFinite(observation.latencyMs) || observation.latencyMs < 0) return unknown('LATENCY_MISSING', 'latencyMs missing');
|
|
88
|
+
return ok(observation.latencyMs <= assertion.value, 'Recorded latency compared to threshold');
|
|
89
|
+
}
|
|
90
|
+
return mk('not_supported', 'UNSUPPORTED_ASSERTION', 'Unsupported assertion');
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
function caseStatus(statuses) {
|
|
94
|
+
for (const s of ['fail', 'error', 'not_supported', 'inconclusive']) if (statuses.includes(s)) return s;
|
|
95
|
+
return 'pass';
|
|
96
|
+
}
|
|
97
|
+
|
|
98
|
+
export function evaluateSuite(suite, observations, provenance = {}) {
|
|
99
|
+
const errors = validateSuite(suite);
|
|
100
|
+
if (errors.length) throw new Error(errors.join('; '));
|
|
101
|
+
if (!observations || typeof observations !== 'object' || !observations.cases || typeof observations.cases !== 'object') throw new Error('observations.cases map is required');
|
|
102
|
+
const all = [];
|
|
103
|
+
let totalWeight = 0, scoredWeight = 0, passingWeight = 0;
|
|
104
|
+
for (const item of suite.cases) {
|
|
105
|
+
const obs = Object.hasOwn(observations.cases, item.id) ? observations.cases[item.id] : undefined;
|
|
106
|
+
if (item.skip === true) { all.push({ id: item.id, title: item.title ?? item.id, status: 'skipped', critical: item.critical === true, sourceType: null, assertions: [] }); continue; }
|
|
107
|
+
const sourceType = SOURCE_TYPES.includes(obs?.sourceType) ? obs.sourceType : (obs ? 'unspecified' : null);
|
|
108
|
+
const details = item.assertions.map(a => {
|
|
109
|
+
let r = evaluateAssertion(a, obs);
|
|
110
|
+
if (r.status === 'pass' && WEAK_SOURCES.has(sourceType)) r = { status: 'inconclusive', reason: 'WEAK_SOURCE', detail: `${sourceType} cannot prove a pass` };
|
|
111
|
+
return { type: a.type, status: r.status, reason: r.reason, detail: r.detail, weight: a.weight ?? 1 };
|
|
112
|
+
});
|
|
113
|
+
for (const d of details) {
|
|
114
|
+
totalWeight += d.weight;
|
|
115
|
+
if (d.status === 'pass' || d.status === 'fail') scoredWeight += d.weight;
|
|
116
|
+
if (d.status === 'pass') passingWeight += d.weight;
|
|
117
|
+
}
|
|
118
|
+
all.push({ id: item.id, title: item.title ?? item.id, status: caseStatus(details.map(d => d.status)), critical: item.critical === true, sourceType, assertions: details });
|
|
119
|
+
}
|
|
120
|
+
const count = s => all.filter(x => x.status === s).length;
|
|
121
|
+
const scoredCases = count('pass') + count('fail');
|
|
122
|
+
const coverage = totalWeight ? scoredWeight / totalWeight : 0;
|
|
123
|
+
const passRate = scoredWeight ? passingWeight / scoredWeight : null;
|
|
124
|
+
const gate = suite.gate ?? {};
|
|
125
|
+
const minPass = gate.minPassRate ?? 1, minCover = gate.minCoverage ?? 1;
|
|
126
|
+
const critFail = all.some(x => x.critical && x.status === 'fail');
|
|
127
|
+
const critUnknown = all.some(x => x.critical && !['pass', 'fail', 'skipped'].includes(x.status));
|
|
128
|
+
const notPassing = all.some(x => ['error', 'not_supported', 'inconclusive'].includes(x.status));
|
|
129
|
+
const releaseGate = (critFail || count('fail') > 0 || (passRate !== null && passRate < minPass)) ? 'NO_GO'
|
|
130
|
+
: (critUnknown || coverage < minCover || passRate === null || notPassing) ? 'REVIEW' : 'GO_CANDIDATE';
|
|
131
|
+
return {
|
|
132
|
+
schemaVersion: 2, kind: 'offline-replay-evaluation', scoringVersion: SCORING_VERSION, suite: suite.name,
|
|
133
|
+
recordedAt: observations.recordedAt ?? null, provenance,
|
|
134
|
+
warning: 'Evaluates supplied observations only; no target agent was executed by EvalSutra.',
|
|
135
|
+
metrics: {
|
|
136
|
+
caseCount: all.length, passed: count('pass'), failed: count('fail'), inconclusive: count('inconclusive'),
|
|
137
|
+
errors: count('error'), skipped: count('skipped'), notSupported: count('not_supported'),
|
|
138
|
+
criticalFailures: all.filter(x => x.critical && x.status === 'fail').length,
|
|
139
|
+
coverage, passRate,
|
|
140
|
+
caseLevel: { scored: scoredCases, passRateCI95: wilson(count('pass'), scoredCases) }
|
|
141
|
+
},
|
|
142
|
+
thresholds: { minPassRate: minPass, minCoverage: minCover }, releaseGate, cases: all
|
|
143
|
+
};
|
|
144
|
+
}
|
|
145
|
+
|
|
146
|
+
// Paired comparison on matched case IDs. Verdicts follow the Regression Firewall: PASS / FAIL / INCONCLUSIVE.
|
|
147
|
+
export function compareReports(baseline, candidate, { minPairs = 5 } = {}) {
|
|
148
|
+
if (!baseline?.cases || !candidate?.cases) throw new Error('Two evaluation reports are required');
|
|
149
|
+
const b = new Map(baseline.cases.map(x => [x.id, x.status]));
|
|
150
|
+
const c = new Map(candidate.cases.map(x => [x.id, x.status]));
|
|
151
|
+
const changes = candidate.cases.map(x => ({
|
|
152
|
+
id: x.id, before: b.get(x.id) ?? 'not_present', after: x.status,
|
|
153
|
+
change: b.get(x.id) === 'pass' && x.status === 'fail' ? 'regression' : b.get(x.id) === 'fail' && x.status === 'pass' ? 'improvement' : b.has(x.id) ? 'unchanged_or_unscored' : 'new_case'
|
|
154
|
+
}));
|
|
155
|
+
for (const id of b.keys()) if (!c.has(id)) changes.push({ id, before: b.get(id), after: 'missing', change: 'removed_case' });
|
|
156
|
+
const regressions = changes.filter(x => x.change === 'regression'), improvements = changes.filter(x => x.change === 'improvement');
|
|
157
|
+
const matched = changes.filter(x => b.has(x.id) && c.has(x.id));
|
|
158
|
+
const scoredPairs = matched.filter(x => ['pass', 'fail'].includes(x.before) && ['pass', 'fail'].includes(x.after)).length;
|
|
159
|
+
const mismatched = changes.some(x => ['new_case', 'removed_case'].includes(x.change));
|
|
160
|
+
const confounders = [];
|
|
161
|
+
if (mismatched) confounders.push('case sets differ');
|
|
162
|
+
if (baseline.suite !== candidate.suite) confounders.push('suite names differ');
|
|
163
|
+
if (scoredPairs < minPairs) confounders.push(`only ${scoredPairs} scored pairs (<${minPairs})`);
|
|
164
|
+
const verdict = regressions.length ? 'FAIL' : confounders.length ? 'INCONCLUSIVE' : 'PASS';
|
|
165
|
+
return {
|
|
166
|
+
schemaVersion: 2, baseline: baseline.suite, candidate: candidate.suite, verdict, confounders,
|
|
167
|
+
pairs: { matched: matched.length, scored: scoredPairs, signTestP: signTestP(regressions.length, improvements.length) },
|
|
168
|
+
regressions, improvements, changes
|
|
169
|
+
};
|
|
170
|
+
}
|
|
171
|
+
|
|
172
|
+
// Exact two-sided sign test on discordant pairs (null: regression and improvement equally likely).
|
|
173
|
+
function signTestP(r, i) {
|
|
174
|
+
const n = r + i;
|
|
175
|
+
if (!n) return null;
|
|
176
|
+
const k = Math.min(r, i);
|
|
177
|
+
let tail = 0, coef = 1;
|
|
178
|
+
for (let j = 0; j <= k; j++) { tail += coef; coef = coef * (n - j) / (j + 1); }
|
|
179
|
+
return Math.min(1, 2 * tail / 2 ** n);
|
|
180
|
+
}
|
|
181
|
+
|
|
182
|
+
const pct = v => v === null || v === undefined ? 'not measured' : `${(v * 100).toFixed(1)}%`;
|
|
183
|
+
export function makeMarkdown(r) {
|
|
184
|
+
const ci = r.metrics.caseLevel?.passRateCI95;
|
|
185
|
+
const rows = r.cases.map(c => `| ${c.id} | ${c.status} | ${c.critical ? 'yes' : 'no'} | ${c.sourceType ?? 'none'} |`).join('\n');
|
|
186
|
+
return `# EvalSutra evaluation report\n\nSuite: **${r.suite}** \nMode: **offline replay** \nScoring: **${r.scoringVersion ?? 'scoring-v1'}** \nGate: **${r.releaseGate}** \nPass rate (scored assertions only): **${pct(r.metrics.passRate)}** \nEvidence coverage: **${pct(r.metrics.coverage)}** \n${ci ? `Case-level pass rate 95% Wilson CI: **${pct(ci[0])} – ${pct(ci[1])}** (n=${r.metrics.caseLevel.scored}) \n` : ''}\n> ${r.warning}\n\n| Case | Result | Critical | Evidence source |\n|---|---|---|---|\n${rows}\n\nA GO_CANDIDATE is not a release authorization. Human review remains required.\n`;
|
|
187
|
+
}
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
import { createHash } from 'node:crypto';
|
|
2
|
+
|
|
3
|
+
const PATTERNS = [
|
|
4
|
+
[/-----BEGIN [A-Z ]*PRIVATE KEY-----[\s\S]*?-----END [A-Z ]*PRIVATE KEY-----/g, '[REDACTED_PRIVATE_KEY]'],
|
|
5
|
+
[/\bgh[pousr]_[A-Za-z0-9]{30,}\b/g, '[REDACTED_GITHUB_TOKEN]'],
|
|
6
|
+
[/\bgithub_pat_[A-Za-z0-9_]{30,}\b/g, '[REDACTED_GITHUB_TOKEN]'],
|
|
7
|
+
[/\bsk-[A-Za-z0-9_-]{20,}\b/g, '[REDACTED_API_KEY]'],
|
|
8
|
+
[/\bAKIA[0-9A-Z]{16}\b/g, '[REDACTED_AWS_KEY]'],
|
|
9
|
+
[/\bBearer\s+[A-Za-z0-9._~+/=-]{20,}/g, 'Bearer [REDACTED]']
|
|
10
|
+
];
|
|
11
|
+
|
|
12
|
+
export function redact(text) {
|
|
13
|
+
return PATTERNS.reduce((s, [re, rep]) => s.replace(re, rep), String(text));
|
|
14
|
+
}
|
|
15
|
+
export const hasSecret = text => redact(text) !== String(text);
|
|
16
|
+
export const redactDeep = v => JSON.parse(redact(JSON.stringify(v)));
|
|
17
|
+
|
|
18
|
+
const stable = v => Array.isArray(v) ? v.map(stable)
|
|
19
|
+
: v && typeof v === 'object' ? Object.fromEntries(Object.keys(v).sort().map(k => [k, stable(v[k])])) : v;
|
|
20
|
+
export const stableStringify = v => JSON.stringify(stable(v));
|
|
21
|
+
export const sha256 = data => createHash('sha256').update(data).digest('hex');
|
|
22
|
+
export const digest = v => sha256(stableStringify(v));
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
// Wilson score interval; returns null when there are no trials (never fabricate certainty).
|
|
2
|
+
export function wilson(successes, n, z = 1.96) {
|
|
3
|
+
if (!n) return null;
|
|
4
|
+
const p = successes / n, z2 = z * z, d = 1 + z2 / n;
|
|
5
|
+
const c = (p + z2 / (2 * n)) / d, h = z * Math.sqrt(p * (1 - p) / n + z2 / (4 * n * n)) / d;
|
|
6
|
+
return [Math.max(0, c - h), Math.min(1, c + h)];
|
|
7
|
+
}
|
|
8
|
+
|
|
9
|
+
// Unbiased pass@k estimator (Chen et al. 2021) for n trials with c correct.
|
|
10
|
+
export function passAtK(n, c, k) {
|
|
11
|
+
if (!Number.isInteger(n) || !Number.isInteger(c) || !Number.isInteger(k) || k < 1 || k > n || c < 0 || c > n) throw new Error('passAtK needs integers 1<=k<=n and 0<=c<=n');
|
|
12
|
+
if (n - c < k) return 1;
|
|
13
|
+
let prod = 1;
|
|
14
|
+
for (let i = n - c + 1; i <= n; i++) prod *= 1 - k / i;
|
|
15
|
+
return 1 - prod;
|
|
16
|
+
}
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
import { hasSecret } from './security.mjs';
|
|
2
|
+
|
|
3
|
+
// Trace format: {events:[{id, type:'tool_call', name, args?, ok?}, {id, type:'final', text}]}
|
|
4
|
+
const SUCCESS_CLAIM = /\b(success(ful(ly)?)?|done|completed?|fixed|all tests pass(ed)?)\b/i;
|
|
5
|
+
|
|
6
|
+
export function analyzeTrace(trace, { forbidden = [], allowed = null, loopLimit = 3 } = {}) {
|
|
7
|
+
if (!Array.isArray(trace?.events)) throw new Error('trace.events array is required');
|
|
8
|
+
const findings = [];
|
|
9
|
+
const add = (code, event, detail) => findings.push({ code, evidence: event?.id ?? null, detail });
|
|
10
|
+
const calls = trace.events.filter(e => e.type === 'tool_call');
|
|
11
|
+
let run = 0, prev = null;
|
|
12
|
+
for (const e of calls) {
|
|
13
|
+
if (forbidden.includes(e.name)) add('FORBIDDEN_TOOL', e, `forbidden tool ${e.name}`);
|
|
14
|
+
if (allowed && !allowed.includes(e.name)) add('UNAUTHORIZED_TOOL', e, `tool ${e.name} not in allowlist`);
|
|
15
|
+
if (hasSecret(JSON.stringify(e.args ?? {}))) add('SECRET_IN_ARGS', e, 'credential-like value in tool arguments');
|
|
16
|
+
const key = `${e.name}:${JSON.stringify(e.args ?? null)}`;
|
|
17
|
+
run = key === prev ? run + 1 : 1; prev = key;
|
|
18
|
+
if (run === loopLimit) add('REDUNDANT_LOOP', e, `${e.name} repeated ${loopLimit}x with identical arguments`);
|
|
19
|
+
}
|
|
20
|
+
const failed = calls.filter(e => e.ok === false);
|
|
21
|
+
const final = [...trace.events].reverse().find(e => e.type === 'final');
|
|
22
|
+
if (!final) add('FINAL_MISSING', null, 'no final event; completion cannot be judged');
|
|
23
|
+
else {
|
|
24
|
+
if (failed.length && SUCCESS_CLAIM.test(final.text ?? '') && failed.at(-1) === calls.at(-1)) add('UNSUPPORTED_SUCCESS_CLAIM', final, 'final text claims success but last tool call failed');
|
|
25
|
+
if (hasSecret(final.text ?? '')) add('SECRET_IN_OUTPUT', final, 'credential-like value in final output');
|
|
26
|
+
}
|
|
27
|
+
if (calls.some(e => e.ok === undefined)) add('RESULT_UNKNOWN', null, 'some tool calls lack ok status');
|
|
28
|
+
return { schemaVersion: 1, calls: calls.length, failedCalls: failed.length, findings, status: findings.some(f => !['RESULT_UNKNOWN', 'FINAL_MISSING'].includes(f.code)) ? 'fail' : findings.length ? 'inconclusive' : 'pass' };
|
|
29
|
+
}
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
import { esc, badge, page } from './theme.mjs';
|
|
2
|
+
|
|
3
|
+
const pct = v => v === null || v === undefined ? 'not measured' : `${(v * 100).toFixed(1)}%`;
|
|
4
|
+
|
|
5
|
+
export function toHtml(r) {
|
|
6
|
+
const m = r.metrics, ci = m.caseLevel?.passRateCI95;
|
|
7
|
+
const stats = [
|
|
8
|
+
[pct(m.passRate), 'pass rate (scored assertions only)'], [pct(m.coverage), 'evidence coverage'],
|
|
9
|
+
[`${m.passed}/${m.caseCount}`, 'cases passed'], [String(m.failed), 'failed'], [String(m.inconclusive + m.errors + m.notSupported), 'inconclusive / error'], [String(m.criticalFailures ?? 0), 'critical failures']
|
|
10
|
+
].map(([v, l]) => `<div class="card"><div class="stat">${esc(v)}<small>${esc(l)}</small></div></div>`).join('');
|
|
11
|
+
const rows = r.cases.map(c => `<tr><th scope="row">${esc(c.id)}</th><td>${badge(c.status)}</td><td>${c.critical ? 'yes' : 'no'}</td><td>${esc(c.sourceType ?? 'none')}</td><td>${c.assertions.map(a => `${esc(a.type)}: ${esc(a.status)}${a.reason && a.reason !== 'OK' ? ` (${esc(a.reason)})` : ''}`).join('<br>')}</td></tr>`).join('');
|
|
12
|
+
const body = `<section class="hero"><p class="tag">Evaluation report</p><h1>${esc(r.suite)}</h1><p>Release gate: ${badge(r.releaseGate)}</p><p>${esc(r.warning)}</p></section>
|
|
13
|
+
<div class="grid">${stats}</div>
|
|
14
|
+
${ci ? `<p>Case-level pass rate 95% Wilson interval: <strong>${pct(ci[0])} – ${pct(ci[1])}</strong> (n=${m.caseLevel.scored}).</p>` : '<p>No scored cases: no interval can be computed.</p>'}
|
|
15
|
+
<h2>Cases</h2><div class="tablewrap"><table><caption>Status is shown as text and symbol.</caption><thead><tr><th>Case</th><th>Status</th><th>Critical</th><th>Evidence source</th><th>Assertions</th></tr></thead><tbody>${rows}</tbody></table></div>
|
|
16
|
+
<h2>Provenance</h2><div class="tablewrap"><table><tbody>${Object.entries(r.provenance ?? {}).map(([k, v]) => `<tr><th scope="row">${esc(k)}</th><td><code>${esc(v)}</code></td></tr>`).join('')}<tr><th scope="row">scoring</th><td><code>${esc(r.scoringVersion ?? 'scoring-v1')}</code></td></tr><tr><th scope="row">reportDigest</th><td><code>${esc(r.reportDigest ?? 'n/a')}</code></td></tr></tbody></table></div>
|
|
17
|
+
<blockquote>GO_CANDIDATE is a recommendation on recorded evidence. It is not deployment approval. Human review remains required.</blockquote>`;
|
|
18
|
+
return page({ title: `EvalSutra report: ${r.suite}`, description: `Offline evaluation report for ${r.suite}`, body });
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
export function toJunit(r) {
|
|
22
|
+
const bad = c => c.status === 'fail' ? '<failure message="assertion failed"/>' : c.status === 'pass' ? '' : c.status === 'skipped' ? '<skipped/>' : `<error message="${esc(c.status)}"/>`;
|
|
23
|
+
const cases = r.cases.map(c => `<testcase classname="${esc(r.suite)}" name="${esc(c.id)}">${bad(c)}</testcase>`).join('');
|
|
24
|
+
const m = r.metrics;
|
|
25
|
+
return `<?xml version="1.0" encoding="UTF-8"?>\n<testsuite name="${esc(r.suite)}" tests="${m.caseCount}" failures="${m.failed}" errors="${m.errors + m.inconclusive + m.notSupported}" skipped="${m.skipped}">${cases}</testsuite>\n`;
|
|
26
|
+
}
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
export const esc = s => String(s).replace(/[&<>"']/g, c => ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' })[c]);
|
|
2
|
+
|
|
3
|
+
// One accessible design system shared by reports, audit and docs. Status is always text + symbol, never color alone.
|
|
4
|
+
export const CSS = `
|
|
5
|
+
:root{--bg:#f7f8fb;--surface:#fff;--ink:#14161f;--muted:#566074;--line:#d9deea;--brand:#4b3fd6;--brand-ink:#fff;--ok:#0b6b3a;--ok-bg:#e3f6ea;--bad:#a4161a;--bad-bg:#fde8e8;--warn:#7a4b00;--warn-bg:#fff2d6;--info:#124e8a;--info-bg:#e4f0fc;--code:#eef0f7;--radius:14px;--shadow:0 1px 2px rgba(20,22,31,.06),0 8px 24px rgba(20,22,31,.06)}
|
|
6
|
+
@media (prefers-color-scheme:dark){:root{--bg:#0e1017;--surface:#161a24;--ink:#eceff7;--muted:#9aa4ba;--line:#2a3042;--brand:#8f86ff;--brand-ink:#0e1017;--ok:#6fe0a0;--ok-bg:#12301f;--bad:#ff8d90;--bad-bg:#3a1618;--warn:#ffd27a;--warn-bg:#3a2a0a;--info:#8cc4ff;--info-bg:#10294a;--code:#1d2232;--shadow:none}}
|
|
7
|
+
*{box-sizing:border-box}html{scroll-behavior:smooth}
|
|
8
|
+
body{margin:0;background:var(--bg);color:var(--ink);font:16px/1.65 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;-webkit-font-smoothing:antialiased}
|
|
9
|
+
a{color:var(--brand)}a:hover{text-decoration-thickness:2px}
|
|
10
|
+
:focus-visible{outline:3px solid var(--brand);outline-offset:2px;border-radius:4px}
|
|
11
|
+
.skip{position:absolute;left:-999px;top:0;background:var(--brand);color:var(--brand-ink);padding:.6rem 1rem;z-index:9}.skip:focus{left:0}
|
|
12
|
+
header.site{border-bottom:1px solid var(--line);background:var(--surface);position:sticky;top:0;z-index:5}
|
|
13
|
+
header.site div{max-width:72rem;margin:auto;padding:.7rem 1.25rem;display:flex;gap:1rem;align-items:center;flex-wrap:wrap}
|
|
14
|
+
header.site strong{font-size:1.05rem;letter-spacing:.2px}header.site nav{display:flex;gap:.25rem 1rem;flex-wrap:wrap;margin-left:auto}header.site nav a{text-decoration:none;color:var(--muted);font-size:.92rem;padding:.2rem .1rem}header.site nav a[aria-current=page],header.site nav a:hover{color:var(--ink);box-shadow:0 2px 0 var(--brand)}
|
|
15
|
+
main{max-width:72rem;margin:0 auto;padding:2rem 1.25rem 4rem}
|
|
16
|
+
.hero{background:linear-gradient(135deg,var(--brand),#7b5cff 60%,#00a3a3);color:#fff;border-radius:calc(var(--radius) + 6px);padding:2.4rem 2rem;box-shadow:var(--shadow);margin-bottom:2rem}
|
|
17
|
+
.hero h1{margin:.1rem 0 .4rem;font-size:clamp(1.7rem,4vw,2.6rem);line-height:1.15}.hero p{margin:.2rem 0;max-width:46rem;opacity:.95}
|
|
18
|
+
.hero .tag{font-weight:600;letter-spacing:.3px}
|
|
19
|
+
h1,h2,h3{line-height:1.25}h2{margin-top:2.4rem;padding-bottom:.35rem;border-bottom:1px solid var(--line)}h3{margin-top:1.6rem}
|
|
20
|
+
.grid{display:grid;gap:1rem;grid-template-columns:repeat(auto-fit,minmax(15rem,1fr));margin:1rem 0}
|
|
21
|
+
.card{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:1.1rem 1.2rem;box-shadow:var(--shadow)}
|
|
22
|
+
.card h3{margin:.1rem 0 .4rem;font-size:1.05rem}.card p{margin:.3rem 0;color:var(--muted)}
|
|
23
|
+
.stat{font-size:2rem;font-weight:700;line-height:1.1}.stat small{display:block;font-size:.8rem;font-weight:500;color:var(--muted);margin-top:.3rem}
|
|
24
|
+
table{border-collapse:collapse;width:100%;background:var(--surface);border:1px solid var(--line);border-radius:10px;overflow:hidden;font-size:.95rem}
|
|
25
|
+
caption{text-align:left;padding:.5rem 0;color:var(--muted);font-size:.9rem}
|
|
26
|
+
th,td{padding:.55rem .75rem;border-bottom:1px solid var(--line);text-align:left;vertical-align:top}thead th{background:var(--code);font-size:.85rem;text-transform:uppercase;letter-spacing:.4px}tbody tr:last-child td,tbody tr:last-child th{border-bottom:0}
|
|
27
|
+
.tablewrap{overflow-x:auto;margin:1rem 0}
|
|
28
|
+
code,pre{font:.9em/1.5 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}code{background:var(--code);padding:.1rem .35rem;border-radius:5px}
|
|
29
|
+
pre{background:var(--code);padding:1rem;border-radius:10px;overflow-x:auto;border:1px solid var(--line)}pre code{background:none;padding:0}
|
|
30
|
+
blockquote{margin:1rem 0;padding:.6rem 1rem;border-left:4px solid var(--brand);background:var(--info-bg);border-radius:0 10px 10px 0}
|
|
31
|
+
.badge{display:inline-flex;align-items:center;gap:.3rem;padding:.12rem .6rem;border-radius:999px;font-size:.82rem;font-weight:600;border:1px solid currentColor;white-space:nowrap}
|
|
32
|
+
.b-pass,.b-go_candidate{color:var(--ok);background:var(--ok-bg)}.b-fail,.b-no_go{color:var(--bad);background:var(--bad-bg)}.b-inconclusive,.b-review,.b-error,.b-not_supported,.b-skipped{color:var(--warn);background:var(--warn-bg)}.b-info{color:var(--info);background:var(--info-bg)}
|
|
33
|
+
footer{max-width:72rem;margin:0 auto;padding:1.5rem 1.25rem 3rem;color:var(--muted);font-size:.88rem;border-top:1px solid var(--line)}
|
|
34
|
+
.toc{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:1rem 1.3rem;margin:1rem 0}.toc ul{margin:.3rem 0;padding-left:1.2rem}
|
|
35
|
+
@media (max-width:40rem){header.site nav{margin-left:0}.hero{padding:1.5rem 1.1rem}}
|
|
36
|
+
@media print{header.site,.skip{display:none}body{background:#fff;color:#000}.card,table{box-shadow:none}a{color:#000}}
|
|
37
|
+
@media (prefers-reduced-motion:reduce){html{scroll-behavior:auto}}
|
|
38
|
+
`;
|
|
39
|
+
|
|
40
|
+
const ICON = { pass: '✔', fail: '✖', inconclusive: '?', error: '!', skipped: '–', not_supported: '–', go_candidate: '✔', no_go: '✖', review: '?' };
|
|
41
|
+
export const badge = status => `<span class="badge b-${esc(String(status).toLowerCase())}"><span aria-hidden="true">${ICON[String(status).toLowerCase()] ?? '•'}</span> ${esc(String(status).toUpperCase())}</span>`;
|
|
42
|
+
|
|
43
|
+
export function page({ title, description, body, nav = '', lang = 'en', canonical = '' }) {
|
|
44
|
+
return `<!doctype html>
|
|
45
|
+
<html lang="${lang}"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
46
|
+
<title>${esc(title)}</title><meta name="description" content="${esc(description ?? '')}"><meta name="author" content="Vatsal Shah"><meta name="color-scheme" content="light dark">
|
|
47
|
+
${canonical ? `<link rel="canonical" href="${esc(canonical)}">` : ''}<meta property="og:title" content="${esc(title)}"><meta property="og:description" content="${esc(description ?? '')}"><meta property="og:type" content="website">
|
|
48
|
+
<style>${CSS}</style></head><body><a class="skip" href="#main">Skip to content</a>
|
|
49
|
+
<header class="site"><div><strong>EvalSutra AI</strong>${nav}</div></header>
|
|
50
|
+
<main id="main">${body}</main>
|
|
51
|
+
<footer>EvalSutra AI · Created by Vatsal Shah · Apache-2.0 · Generated offline; no scripts, no trackers, no network calls.</footer></body></html>
|
|
52
|
+
`;
|
|
53
|
+
}
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"minPassRate":0.9,"minCoverage":0.9,"maxInconclusive":0}
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-arena
|
|
3
|
+
description: Compare agents on matched tasks with adequate samples and transparent trade-offs.
|
|
4
|
+
---
|
|
5
|
+
# Comparative Evaluation Analyst — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Compare agents on matched tasks with adequate samples and transparent trade-offs.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): compare. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-audit
|
|
3
|
+
description: Inspect grader logic, evidence authenticity, repeatability, score validity and risk of gaming.
|
|
4
|
+
---
|
|
5
|
+
# Evaluation Integrity Auditor — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Inspect grader logic, evidence authenticity, repeatability, score validity and risk of gaming.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): repro verify, validate. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-bench
|
|
3
|
+
description: Construct controlled suites, version fixtures and validate ground truths.
|
|
4
|
+
---
|
|
5
|
+
# Benchmark Engineer — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Construct controlled suites, version fixtures and validate ground truths.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): init, validate. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-cost
|
|
3
|
+
description: Analyze recorded token, latency, throughput and cost measurements with units and unknowns.
|
|
4
|
+
---
|
|
5
|
+
# Efficiency Evaluator — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Analyze recorded token, latency, throughput and cost measurements with units and unknowns.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): run (max_latency_ms only; token cost not implemented). Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-data
|
|
3
|
+
description: Verify fixture provenance, privacy, licensing and test leakage controls.
|
|
4
|
+
---
|
|
5
|
+
# Benchmark Data Steward — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Verify fixture provenance, privacy, licensing and test leakage controls.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): repro export/verify (hashes), validate. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-drift
|
|
3
|
+
description: Compare controlled baselines and candidates; identify regressions and confounding changes.
|
|
4
|
+
---
|
|
5
|
+
# Regression and Drift Analyst — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Compare controlled baselines and candidates; identify regressions and confounding changes.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): compare. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-explain
|
|
3
|
+
description: Correlate failed assertions with relevant evidence and recommend targeted investigation.
|
|
4
|
+
---
|
|
5
|
+
# Failure Diagnosis Specialist — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Correlate failed assertions with relevant evidence and recommend targeted investigation.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): report, trace analyze. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-guard
|
|
3
|
+
description: Check synthetic prompt injection, tool abuse and secret handling inside authorized isolated fixtures.
|
|
4
|
+
---
|
|
5
|
+
# Permission and Security Evaluator — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Check synthetic prompt injection, tool abuse and secret handling inside authorized isolated fixtures.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): trace analyze --forbid, run (not_contains canary). Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-judge
|
|
3
|
+
description: Define rubric criteria and optional LLM grader bias checks; never pass unsupported findings.
|
|
4
|
+
---
|
|
5
|
+
# Rubric and Judge Calibration Specialist — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Define rubric criteria and optional LLM grader bias checks; never pass unsupported findings.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): none: judge output is labeled model_judge and can never prove a pass. Everything else is manual. See `docs/STATUS.md`.
|