evalsutra 0.0.0-stage → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/evalsutra-arena/SKILL.md +27 -0
- package/.agents/skills/evalsutra-audit/SKILL.md +27 -0
- package/.agents/skills/evalsutra-bench/SKILL.md +27 -0
- package/.agents/skills/evalsutra-cost/SKILL.md +27 -0
- package/.agents/skills/evalsutra-data/SKILL.md +27 -0
- package/.agents/skills/evalsutra-drift/SKILL.md +27 -0
- package/.agents/skills/evalsutra-explain/SKILL.md +27 -0
- package/.agents/skills/evalsutra-guard/SKILL.md +27 -0
- package/.agents/skills/evalsutra-judge/SKILL.md +27 -0
- package/.agents/skills/evalsutra-pluginwatch/SKILL.md +27 -0
- package/.agents/skills/evalsutra-port/SKILL.md +27 -0
- package/.agents/skills/evalsutra-prime/SKILL.md +27 -0
- package/.agents/skills/evalsutra-report/SKILL.md +27 -0
- package/.agents/skills/evalsutra-robust/SKILL.md +27 -0
- package/.agents/skills/evalsutra-task/SKILL.md +27 -0
- package/.agents/skills/evalsutra-trace/SKILL.md +27 -0
- package/.agents/skills/evalsutra-truth/SKILL.md +27 -0
- package/.agents/skills/evalsutra-verify/SKILL.md +27 -0
- package/.claude-plugin/plugin.json +8 -0
- package/CHANGELOG.md +33 -0
- package/CITATION.cff +14 -0
- package/CODE_OF_CONDUCT.md +2 -0
- package/CONTRIBUTING.md +2 -0
- package/LICENSE +202 -0
- package/NOTICE +5 -0
- package/README.md +161 -2
- package/SECURITY.md +17 -0
- package/SUPPORT.md +5 -0
- package/THIRD_PARTY_NOTICES.md +14 -0
- package/docs/ADVANCED_TOOLING_ROADMAP.md +244 -0
- package/docs/ARCHITECTURE.md +17 -0
- package/docs/PLUGIN_AND_SKILL_CATALOG.md +56 -0
- package/docs/STATUS.md +25 -0
- package/docs/master-claude-prompt.md +534 -0
- package/documents/evalsutra/01-executive-summary.md +52 -0
- package/documents/evalsutra/02-functional-specification.md +104 -0
- package/documents/evalsutra/03-architecture.md +78 -0
- package/documents/evalsutra/04-technical-reference.md +150 -0
- package/documents/evalsutra/05-user-guide.md +94 -0
- package/documents/evalsutra/06-skills-catalog.md +52 -0
- package/documents/evalsutra/07-security-and-legal.md +66 -0
- package/documents/evalsutra/08-testing-and-evidence.md +63 -0
- package/documents/evalsutra/09-audit-and-trust.md +54 -0
- package/documents/evalsutra/10-ci-cd-and-release.md +50 -0
- package/documents/evalsutra/11-faq-and-glossary.md +51 -0
- package/documents/evalsutra/12-contributor-handbook.md +67 -0
- package/documents/evalsutra/13-roadmap-and-status.md +48 -0
- package/documents/evalsutra/assets/architecture.svg +21 -0
- package/documents/evalsutra/evidence/README.md +39 -0
- package/documents/evalsutra/evidence/samples/audit/audit-report.html +46 -0
- package/documents/evalsutra/evidence/samples/audit/audit-report.json +317 -0
- package/documents/evalsutra/evidence/samples/audit/audit-report.md +64 -0
- package/documents/evalsutra/evidence/samples/bad/junit.xml +2 -0
- package/documents/evalsutra/evidence/samples/bad/report.html +47 -0
- package/documents/evalsutra/evidence/samples/bad/report.json +153 -0
- package/documents/evalsutra/evidence/samples/bad/report.md +22 -0
- package/documents/evalsutra/evidence/samples/compare-good-vs-bad.json +59 -0
- package/documents/evalsutra/evidence/samples/demo-suite.json +13 -0
- package/documents/evalsutra/evidence/samples/gate-bad-run.json +10 -0
- package/documents/evalsutra/evidence/samples/good/junit.xml +2 -0
- package/documents/evalsutra/evidence/samples/good/report.html +47 -0
- package/documents/evalsutra/evidence/samples/good/report.json +153 -0
- package/documents/evalsutra/evidence/samples/good/report.md +22 -0
- package/documents/evalsutra/evidence/samples/trace-analysis.json +33 -0
- package/documents/evalsutra/evidence/test-run.txt +71 -0
- package/documents/evalsutra/evidence/test-summary.json +11 -0
- package/documents/evalsutra/html/01-executive-summary.html +57 -0
- package/documents/evalsutra/html/02-functional-specification.html +75 -0
- package/documents/evalsutra/html/03-architecture.html +60 -0
- package/documents/evalsutra/html/04-technical-reference.html +109 -0
- package/documents/evalsutra/html/05-user-guide.html +72 -0
- package/documents/evalsutra/html/06-skills-catalog.html +54 -0
- package/documents/evalsutra/html/07-security-and-legal.html +63 -0
- package/documents/evalsutra/html/08-testing-and-evidence.html +61 -0
- package/documents/evalsutra/html/09-audit-and-trust.html +61 -0
- package/documents/evalsutra/html/10-ci-cd-and-release.html +61 -0
- package/documents/evalsutra/html/11-faq-and-glossary.html +57 -0
- package/documents/evalsutra/html/12-contributor-handbook.html +71 -0
- package/documents/evalsutra/html/13-roadmap-and-status.html +51 -0
- package/documents/evalsutra/html/assets/architecture.svg +21 -0
- package/documents/evalsutra/html/index.html +42 -0
- package/examples/demo-observations-bad.json +7 -0
- package/examples/demo-observations.json +7 -0
- package/examples/observations.json +10 -0
- package/examples/output/report.json +99 -0
- package/examples/output/report.md +18 -0
- package/examples/suite.json +11 -0
- package/examples/traces/bad-trace.json +5 -0
- package/llms.txt +25 -0
- package/package.json +19 -4
- package/packages/audit/audit.mjs +179 -0
- package/packages/cli/evalsutra.mjs +151 -0
- package/packages/cli/install.mjs +68 -0
- package/packages/core/capsule.mjs +31 -0
- package/packages/core/evaluate.mjs +187 -0
- package/packages/core/security.mjs +22 -0
- package/packages/core/stats.mjs +16 -0
- package/packages/core/trace.mjs +29 -0
- package/packages/reporters/index.mjs +26 -0
- package/packages/reporters/theme.mjs +53 -0
- package/policies/README.md +3 -0
- package/policies/default.json +1 -0
- package/skills/evalsutra-arena/SKILL.md +27 -0
- package/skills/evalsutra-audit/SKILL.md +27 -0
- package/skills/evalsutra-bench/SKILL.md +27 -0
- package/skills/evalsutra-cost/SKILL.md +27 -0
- package/skills/evalsutra-data/SKILL.md +27 -0
- package/skills/evalsutra-drift/SKILL.md +27 -0
- package/skills/evalsutra-explain/SKILL.md +27 -0
- package/skills/evalsutra-guard/SKILL.md +27 -0
- package/skills/evalsutra-judge/SKILL.md +27 -0
- package/skills/evalsutra-pluginwatch/SKILL.md +27 -0
- package/skills/evalsutra-port/SKILL.md +27 -0
- package/skills/evalsutra-prime/SKILL.md +27 -0
- package/skills/evalsutra-report/SKILL.md +27 -0
- package/skills/evalsutra-robust/SKILL.md +27 -0
- package/skills/evalsutra-task/SKILL.md +27 -0
- package/skills/evalsutra-trace/SKILL.md +27 -0
- package/skills/evalsutra-truth/SKILL.md +27 -0
- package/skills/evalsutra-verify/SKILL.md +27 -0
- package/suites/demo.json +13 -0
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
<!doctype html>
|
|
2
|
+
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
3
|
+
<title>3. Architecture · EvalSutra AI</title><meta name="description" content="3. Architecture: EvalSutra AI documentation by Vatsal Shah."><meta name="author" content="Vatsal Shah"><meta name="color-scheme" content="light dark">
|
|
4
|
+
<meta property="og:title" content="3. Architecture · EvalSutra AI"><meta property="og:description" content="3. Architecture: EvalSutra AI documentation by Vatsal Shah."><meta property="og:type" content="website">
|
|
5
|
+
<style>
|
|
6
|
+
:root{--bg:#f7f8fb;--surface:#fff;--ink:#14161f;--muted:#566074;--line:#d9deea;--brand:#4b3fd6;--brand-ink:#fff;--ok:#0b6b3a;--ok-bg:#e3f6ea;--bad:#a4161a;--bad-bg:#fde8e8;--warn:#7a4b00;--warn-bg:#fff2d6;--info:#124e8a;--info-bg:#e4f0fc;--code:#eef0f7;--radius:14px;--shadow:0 1px 2px rgba(20,22,31,.06),0 8px 24px rgba(20,22,31,.06)}
|
|
7
|
+
@media (prefers-color-scheme:dark){:root{--bg:#0e1017;--surface:#161a24;--ink:#eceff7;--muted:#9aa4ba;--line:#2a3042;--brand:#8f86ff;--brand-ink:#0e1017;--ok:#6fe0a0;--ok-bg:#12301f;--bad:#ff8d90;--bad-bg:#3a1618;--warn:#ffd27a;--warn-bg:#3a2a0a;--info:#8cc4ff;--info-bg:#10294a;--code:#1d2232;--shadow:none}}
|
|
8
|
+
*{box-sizing:border-box}html{scroll-behavior:smooth}
|
|
9
|
+
body{margin:0;background:var(--bg);color:var(--ink);font:16px/1.65 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;-webkit-font-smoothing:antialiased}
|
|
10
|
+
a{color:var(--brand)}a:hover{text-decoration-thickness:2px}
|
|
11
|
+
:focus-visible{outline:3px solid var(--brand);outline-offset:2px;border-radius:4px}
|
|
12
|
+
.skip{position:absolute;left:-999px;top:0;background:var(--brand);color:var(--brand-ink);padding:.6rem 1rem;z-index:9}.skip:focus{left:0}
|
|
13
|
+
header.site{border-bottom:1px solid var(--line);background:var(--surface);position:sticky;top:0;z-index:5}
|
|
14
|
+
header.site div{max-width:72rem;margin:auto;padding:.7rem 1.25rem;display:flex;gap:1rem;align-items:center;flex-wrap:wrap}
|
|
15
|
+
header.site strong{font-size:1.05rem;letter-spacing:.2px}header.site nav{display:flex;gap:.25rem 1rem;flex-wrap:wrap;margin-left:auto}header.site nav a{text-decoration:none;color:var(--muted);font-size:.92rem;padding:.2rem .1rem}header.site nav a[aria-current=page],header.site nav a:hover{color:var(--ink);box-shadow:0 2px 0 var(--brand)}
|
|
16
|
+
main{max-width:72rem;margin:0 auto;padding:2rem 1.25rem 4rem}
|
|
17
|
+
.hero{background:linear-gradient(135deg,var(--brand),#7b5cff 60%,#00a3a3);color:#fff;border-radius:calc(var(--radius) + 6px);padding:2.4rem 2rem;box-shadow:var(--shadow);margin-bottom:2rem}
|
|
18
|
+
.hero h1{margin:.1rem 0 .4rem;font-size:clamp(1.7rem,4vw,2.6rem);line-height:1.15}.hero p{margin:.2rem 0;max-width:46rem;opacity:.95}
|
|
19
|
+
.hero .tag{font-weight:600;letter-spacing:.3px}
|
|
20
|
+
h1,h2,h3{line-height:1.25}h2{margin-top:2.4rem;padding-bottom:.35rem;border-bottom:1px solid var(--line)}h3{margin-top:1.6rem}
|
|
21
|
+
.grid{display:grid;gap:1rem;grid-template-columns:repeat(auto-fit,minmax(15rem,1fr));margin:1rem 0}
|
|
22
|
+
.card{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:1.1rem 1.2rem;box-shadow:var(--shadow)}
|
|
23
|
+
.card h3{margin:.1rem 0 .4rem;font-size:1.05rem}.card p{margin:.3rem 0;color:var(--muted)}
|
|
24
|
+
.stat{font-size:2rem;font-weight:700;line-height:1.1}.stat small{display:block;font-size:.8rem;font-weight:500;color:var(--muted);margin-top:.3rem}
|
|
25
|
+
table{border-collapse:collapse;width:100%;background:var(--surface);border:1px solid var(--line);border-radius:10px;overflow:hidden;font-size:.95rem}
|
|
26
|
+
caption{text-align:left;padding:.5rem 0;color:var(--muted);font-size:.9rem}
|
|
27
|
+
th,td{padding:.55rem .75rem;border-bottom:1px solid var(--line);text-align:left;vertical-align:top}thead th{background:var(--code);font-size:.85rem;text-transform:uppercase;letter-spacing:.4px}tbody tr:last-child td,tbody tr:last-child th{border-bottom:0}
|
|
28
|
+
.tablewrap{overflow-x:auto;margin:1rem 0}
|
|
29
|
+
code,pre{font:.9em/1.5 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}code{background:var(--code);padding:.1rem .35rem;border-radius:5px}
|
|
30
|
+
pre{background:var(--code);padding:1rem;border-radius:10px;overflow-x:auto;border:1px solid var(--line)}pre code{background:none;padding:0}
|
|
31
|
+
blockquote{margin:1rem 0;padding:.6rem 1rem;border-left:4px solid var(--brand);background:var(--info-bg);border-radius:0 10px 10px 0}
|
|
32
|
+
.badge{display:inline-flex;align-items:center;gap:.3rem;padding:.12rem .6rem;border-radius:999px;font-size:.82rem;font-weight:600;border:1px solid currentColor;white-space:nowrap}
|
|
33
|
+
.b-pass,.b-go_candidate{color:var(--ok);background:var(--ok-bg)}.b-fail,.b-no_go{color:var(--bad);background:var(--bad-bg)}.b-inconclusive,.b-review,.b-error,.b-not_supported,.b-skipped{color:var(--warn);background:var(--warn-bg)}.b-info{color:var(--info);background:var(--info-bg)}
|
|
34
|
+
footer{max-width:72rem;margin:0 auto;padding:1.5rem 1.25rem 3rem;color:var(--muted);font-size:.88rem;border-top:1px solid var(--line)}
|
|
35
|
+
.toc{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:1rem 1.3rem;margin:1rem 0}.toc ul{margin:.3rem 0;padding-left:1.2rem}
|
|
36
|
+
@media (max-width:40rem){header.site nav{margin-left:0}.hero{padding:1.5rem 1.1rem}}
|
|
37
|
+
@media print{header.site,.skip{display:none}body{background:#fff;color:#000}.card,table{box-shadow:none}a{color:#000}}
|
|
38
|
+
@media (prefers-reduced-motion:reduce){html{scroll-behavior:auto}}
|
|
39
|
+
</style></head><body><a class="skip" href="#main">Skip to content</a>
|
|
40
|
+
<header class="site"><div><strong>EvalSutra AI</strong><nav aria-label="Documentation"><a href="01-executive-summary.html">Executive summary</a><a href="02-functional-specification.html">Functional specification</a><a href="03-architecture.html" aria-current="page">Architecture</a><a href="04-technical-reference.html">Technical reference</a><a href="05-user-guide.html">User guide</a><a href="06-skills-catalog.html">Skills catalog</a><a href="07-security-and-legal.html">Security and legal</a><a href="index.html">All docs</a></nav></div></header>
|
|
41
|
+
<main id="main"><article><h1 id="3-architecture">3. Architecture</h1><nav class="toc" aria-label="On this page"><strong>On this page</strong><ul><li><a href="#context-c4-level-1">Context (C4 level 1)</a></li><li><a href="#containers-and-components-c4-levels-2-and-3">Containers and components (C4 levels 2 and 3)</a></li><li><a href="#data-flow">Data flow</a></li><li><a href="#key-design-decisions-adrs">Key design decisions (ADRs)</a></li><li><a href="#failure-modes">Failure modes</a></li><li><a href="#extension-points">Extension points</a></li><li><a href="#deployment-topology">Deployment topology</a></li><li><a href="#observability">Observability</a></li></ul></nav>
|
|
42
|
+
<p><img src="assets/architecture.svg" alt="EvalSutra architecture: CLI over validate, deterministic core, reporters, repro capsule, trace analysis, self-audit and skills" loading="lazy" style="max-width:100%;height:auto"></p>
|
|
43
|
+
<h2 id="context-c4-level-1">Context (C4 level 1)</h2>
|
|
44
|
+
<div class="tablewrap"><table><thead><tr><th>Actor</th><th>Interaction</th></tr></thead><tbody><tr><td>Developer or CI job</td><td>runs <code>evalsutra</code> with a suite and observations, reads reports and exit code</td></tr><tr><td>AI IDE (Claude Code, Cursor, Codex, Gemini CLI, Copilot)</td><td>loads installed <code>SKILL.md</code> files and uses its own AI to follow them</td></tr><tr><td>Target agent</td><td><strong>not touched.</strong> It produces observations elsewhere</td></tr><tr><td>Filesystem</td><td>the only external system EvalSutra reads and writes</td></tr></tbody></table></div>
|
|
45
|
+
<p>There is no network, database, queue, server or model call.</p>
|
|
46
|
+
<h2 id="containers-and-components-c4-levels-2-and-3">Containers and components (C4 levels 2 and 3)</h2>
|
|
47
|
+
<div class="tablewrap"><table><thead><tr><th>Component</th><th>Path</th><th>Responsibility</th></tr></thead><tbody><tr><td>CLI</td><td><code>packages/cli/evalsutra.mjs</code></td><td>argument parsing, exit codes, file I/O</td></tr><tr><td>Installer</td><td><code>packages/cli/install.mjs</code></td><td>skill copy, lock file, traversal and symlink guard</td></tr><tr><td>Evaluator</td><td><code>packages/core/evaluate.mjs</code></td><td>suite validation, assertions, scoring, gate, comparison</td></tr><tr><td>Statistics</td><td><code>packages/core/stats.mjs</code></td><td>Wilson interval, pass@k</td></tr><tr><td>Security helpers</td><td><code>packages/core/security.mjs</code></td><td>secret redaction, stable JSON, SHA-256 digests</td></tr><tr><td>Trace analyser</td><td><code>packages/core/trace.mjs</code></td><td>tool trace findings</td></tr><tr><td>Repro capsule</td><td><code>packages/core/capsule.mjs</code></td><td>export and verify</td></tr><tr><td>Reporters</td><td><code>packages/reporters/</code></td><td>HTML, JUnit; <code>theme.mjs</code> is the shared design system</td></tr><tr><td>Self-audit</td><td><code>packages/audit/audit.mjs</code></td><td>29 checks, MD and HTML audit report</td></tr><tr><td>Skills</td><td><code>.agents/skills/</code>, <code>skills/</code></td><td>18 instruction files, canonical and plugin copy</td></tr><tr><td>Doc and evidence builders</td><td><code>scripts/build-docs.mjs</code>, <code>scripts/build-evidence.mjs</code></td><td>this documentation and its proof</td></tr></tbody></table></div>
|
|
48
|
+
<h2 id="data-flow">Data flow</h2>
|
|
49
|
+
<ol><li>CLI reads suite and observations (<code>readJson</code>, errors become exit code 1 or 4).</li><li><code>validateSuite</code> rejects bad ids, unknown assertion types, unsafe regex, bad weights, bad thresholds.</li><li>For each case and assertion, a <strong>pure function</strong> returns <code>{status, reason, detail}</code>. No assertion can run code from the fixture.</li><li>Weak evidence sources downgrade passes to inconclusive.</li><li>Weights give pass rate and coverage; counts give the gate.</li><li>Digests of suite and observations, tool and Node versions go into <code>provenance</code>.</li><li>Reporters render JSON, Markdown, HTML, JUnit.</li><li><code>compare</code>, <code>gate</code>, <code>repro</code> and <code>audit</code> consume the JSON report.</li></ol>
|
|
50
|
+
<h2 id="key-design-decisions-adrs">Key design decisions (ADRs)</h2>
|
|
51
|
+
<div class="tablewrap"><table><thead><tr><th>ID</th><th>Decision</th><th>Reason</th><th>Trade-off</th></tr></thead><tbody><tr><td>ADR-1</td><td>Plain JavaScript ES modules, no build step</td><td><code>npx</code> and clone both run immediately; nothing to compile or break</td><td>no static types; covered by tests instead</td></tr><tr><td>ADR-2</td><td>Zero dependencies</td><td>no supply-chain risk, tiny audit surface</td><td>some code (Markdown converter, Wilson) written in-house</td></tr><tr><td>ADR-3</td><td>Missing evidence is inconclusive</td><td>prevents score inflation</td><td>more REVIEW results until evidence is complete</td></tr><tr><td>ADR-4</td><td>Model judges cannot prove a pass</td><td>judge output is an opinion</td><td>LLM-graded rubrics cannot gate alone</td></tr><tr><td>ADR-5</td><td>Replay runner only</td><td>live execution needs sandbox and consent design</td><td>users must record observations elsewhere</td></tr><tr><td>ADR-6</td><td>Regex is length-capped and screened, not RE2</td><td>no native dependency</td><td>heuristic; upgrade if untrusted suites become common (marked <code>shortcut:</code> in code)</td></tr><tr><td>ADR-7</td><td>Capsule is hashed, not signed</td><td>signing needs key management</td><td>detects accidents, not forgers</td></tr><tr><td>ADR-8</td><td>Reports contain no script</td><td>safe to open anywhere</td><td>no interactive charts yet</td></tr></tbody></table></div>
|
|
52
|
+
<h2 id="failure-modes">Failure modes</h2>
|
|
53
|
+
<div class="tablewrap"><table><thead><tr><th>Failure</th><th>Behaviour</th></tr></thead><tbody><tr><td>Unreadable or invalid JSON</td><td>exit 1 with the file path and parser message</td></tr><tr><td>Invalid suite</td><td>exit 4 listing every error</td></tr><tr><td>Missing observation or field</td><td>inconclusive with reason code</td></tr><tr><td>Unsafe regex</td><td><code>error</code> status, never executed</td></tr><tr><td>Output folder not writable</td><td>exit 1</td></tr><tr><td>Install target outside project</td><td>refused, exit 1</td></tr><tr><td>Modified skill at uninstall</td><td>kept and reported</td></tr></tbody></table></div>
|
|
54
|
+
<h2 id="extension-points">Extension points</h2>
|
|
55
|
+
<p>Add an assertion type in <code>evaluate.mjs</code> (validate, evaluate, test, audit row). Add a reporter in <code>packages/reporters/</code>. Add a skill folder under <code>.agents/skills/</code> and run the sync (see <a href="12-contributor-handbook.html">Contributor handbook</a>). Planned adapters (OpenTelemetry, Promptfoo, Inspect AI, Harbor, MCP Inspector) are described in <a href="13-roadmap-and-status.html">Roadmap and status</a> and are <strong>not implemented</strong>.</p>
|
|
56
|
+
<h2 id="deployment-topology">Deployment topology</h2>
|
|
57
|
+
<p>A single Node process on the user's machine or CI runner. Distribution: npm (<code>npx evalsutra</code>) and GitHub source (<code>npx github:shahvatsalofficial/evalsutra</code>). Nothing is hosted.</p>
|
|
58
|
+
<h2 id="observability">Observability</h2>
|
|
59
|
+
<p>Output is the report. Exit codes are the machine interface. There is no telemetry.</p></article></main>
|
|
60
|
+
<footer>EvalSutra AI · Created by Vatsal Shah · Apache-2.0 · Generated offline; no scripts, no trackers, no network calls.</footer></body></html>
|
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
<!doctype html>
|
|
2
|
+
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
3
|
+
<title>4. Technical reference · EvalSutra AI</title><meta name="description" content="4. Technical reference: EvalSutra AI documentation by Vatsal Shah."><meta name="author" content="Vatsal Shah"><meta name="color-scheme" content="light dark">
|
|
4
|
+
<meta property="og:title" content="4. Technical reference · EvalSutra AI"><meta property="og:description" content="4. Technical reference: EvalSutra AI documentation by Vatsal Shah."><meta property="og:type" content="website">
|
|
5
|
+
<style>
|
|
6
|
+
:root{--bg:#f7f8fb;--surface:#fff;--ink:#14161f;--muted:#566074;--line:#d9deea;--brand:#4b3fd6;--brand-ink:#fff;--ok:#0b6b3a;--ok-bg:#e3f6ea;--bad:#a4161a;--bad-bg:#fde8e8;--warn:#7a4b00;--warn-bg:#fff2d6;--info:#124e8a;--info-bg:#e4f0fc;--code:#eef0f7;--radius:14px;--shadow:0 1px 2px rgba(20,22,31,.06),0 8px 24px rgba(20,22,31,.06)}
|
|
7
|
+
@media (prefers-color-scheme:dark){:root{--bg:#0e1017;--surface:#161a24;--ink:#eceff7;--muted:#9aa4ba;--line:#2a3042;--brand:#8f86ff;--brand-ink:#0e1017;--ok:#6fe0a0;--ok-bg:#12301f;--bad:#ff8d90;--bad-bg:#3a1618;--warn:#ffd27a;--warn-bg:#3a2a0a;--info:#8cc4ff;--info-bg:#10294a;--code:#1d2232;--shadow:none}}
|
|
8
|
+
*{box-sizing:border-box}html{scroll-behavior:smooth}
|
|
9
|
+
body{margin:0;background:var(--bg);color:var(--ink);font:16px/1.65 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;-webkit-font-smoothing:antialiased}
|
|
10
|
+
a{color:var(--brand)}a:hover{text-decoration-thickness:2px}
|
|
11
|
+
:focus-visible{outline:3px solid var(--brand);outline-offset:2px;border-radius:4px}
|
|
12
|
+
.skip{position:absolute;left:-999px;top:0;background:var(--brand);color:var(--brand-ink);padding:.6rem 1rem;z-index:9}.skip:focus{left:0}
|
|
13
|
+
header.site{border-bottom:1px solid var(--line);background:var(--surface);position:sticky;top:0;z-index:5}
|
|
14
|
+
header.site div{max-width:72rem;margin:auto;padding:.7rem 1.25rem;display:flex;gap:1rem;align-items:center;flex-wrap:wrap}
|
|
15
|
+
header.site strong{font-size:1.05rem;letter-spacing:.2px}header.site nav{display:flex;gap:.25rem 1rem;flex-wrap:wrap;margin-left:auto}header.site nav a{text-decoration:none;color:var(--muted);font-size:.92rem;padding:.2rem .1rem}header.site nav a[aria-current=page],header.site nav a:hover{color:var(--ink);box-shadow:0 2px 0 var(--brand)}
|
|
16
|
+
main{max-width:72rem;margin:0 auto;padding:2rem 1.25rem 4rem}
|
|
17
|
+
.hero{background:linear-gradient(135deg,var(--brand),#7b5cff 60%,#00a3a3);color:#fff;border-radius:calc(var(--radius) + 6px);padding:2.4rem 2rem;box-shadow:var(--shadow);margin-bottom:2rem}
|
|
18
|
+
.hero h1{margin:.1rem 0 .4rem;font-size:clamp(1.7rem,4vw,2.6rem);line-height:1.15}.hero p{margin:.2rem 0;max-width:46rem;opacity:.95}
|
|
19
|
+
.hero .tag{font-weight:600;letter-spacing:.3px}
|
|
20
|
+
h1,h2,h3{line-height:1.25}h2{margin-top:2.4rem;padding-bottom:.35rem;border-bottom:1px solid var(--line)}h3{margin-top:1.6rem}
|
|
21
|
+
.grid{display:grid;gap:1rem;grid-template-columns:repeat(auto-fit,minmax(15rem,1fr));margin:1rem 0}
|
|
22
|
+
.card{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:1.1rem 1.2rem;box-shadow:var(--shadow)}
|
|
23
|
+
.card h3{margin:.1rem 0 .4rem;font-size:1.05rem}.card p{margin:.3rem 0;color:var(--muted)}
|
|
24
|
+
.stat{font-size:2rem;font-weight:700;line-height:1.1}.stat small{display:block;font-size:.8rem;font-weight:500;color:var(--muted);margin-top:.3rem}
|
|
25
|
+
table{border-collapse:collapse;width:100%;background:var(--surface);border:1px solid var(--line);border-radius:10px;overflow:hidden;font-size:.95rem}
|
|
26
|
+
caption{text-align:left;padding:.5rem 0;color:var(--muted);font-size:.9rem}
|
|
27
|
+
th,td{padding:.55rem .75rem;border-bottom:1px solid var(--line);text-align:left;vertical-align:top}thead th{background:var(--code);font-size:.85rem;text-transform:uppercase;letter-spacing:.4px}tbody tr:last-child td,tbody tr:last-child th{border-bottom:0}
|
|
28
|
+
.tablewrap{overflow-x:auto;margin:1rem 0}
|
|
29
|
+
code,pre{font:.9em/1.5 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}code{background:var(--code);padding:.1rem .35rem;border-radius:5px}
|
|
30
|
+
pre{background:var(--code);padding:1rem;border-radius:10px;overflow-x:auto;border:1px solid var(--line)}pre code{background:none;padding:0}
|
|
31
|
+
blockquote{margin:1rem 0;padding:.6rem 1rem;border-left:4px solid var(--brand);background:var(--info-bg);border-radius:0 10px 10px 0}
|
|
32
|
+
.badge{display:inline-flex;align-items:center;gap:.3rem;padding:.12rem .6rem;border-radius:999px;font-size:.82rem;font-weight:600;border:1px solid currentColor;white-space:nowrap}
|
|
33
|
+
.b-pass,.b-go_candidate{color:var(--ok);background:var(--ok-bg)}.b-fail,.b-no_go{color:var(--bad);background:var(--bad-bg)}.b-inconclusive,.b-review,.b-error,.b-not_supported,.b-skipped{color:var(--warn);background:var(--warn-bg)}.b-info{color:var(--info);background:var(--info-bg)}
|
|
34
|
+
footer{max-width:72rem;margin:0 auto;padding:1.5rem 1.25rem 3rem;color:var(--muted);font-size:.88rem;border-top:1px solid var(--line)}
|
|
35
|
+
.toc{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:1rem 1.3rem;margin:1rem 0}.toc ul{margin:.3rem 0;padding-left:1.2rem}
|
|
36
|
+
@media (max-width:40rem){header.site nav{margin-left:0}.hero{padding:1.5rem 1.1rem}}
|
|
37
|
+
@media print{header.site,.skip{display:none}body{background:#fff;color:#000}.card,table{box-shadow:none}a{color:#000}}
|
|
38
|
+
@media (prefers-reduced-motion:reduce){html{scroll-behavior:auto}}
|
|
39
|
+
</style></head><body><a class="skip" href="#main">Skip to content</a>
|
|
40
|
+
<header class="site"><div><strong>EvalSutra AI</strong><nav aria-label="Documentation"><a href="01-executive-summary.html">Executive summary</a><a href="02-functional-specification.html">Functional specification</a><a href="03-architecture.html">Architecture</a><a href="04-technical-reference.html" aria-current="page">Technical reference</a><a href="05-user-guide.html">User guide</a><a href="06-skills-catalog.html">Skills catalog</a><a href="07-security-and-legal.html">Security and legal</a><a href="index.html">All docs</a></nav></div></header>
|
|
41
|
+
<main id="main"><article><h1 id="4-technical-reference">4. Technical reference</h1><nav class="toc" aria-label="On this page"><strong>On this page</strong><ul><li><a href="#requirements">Requirements</a></li><li><a href="#install-and-run">Install and run</a></li><li><a href="#exit-codes">Exit codes</a></li><li><a href="#commands">Commands</a></li><li><a href="#suite-format">Suite format</a></li><li><a href="#observations-format">Observations format</a></li><li><a href="#trace-format">Trace format</a></li><li><a href="#report-format-report-json-schemaversion-2">Report format (report.json, schemaVersion 2)</a></li><li><a href="#policy-file-gate-policy">Policy file (gate --policy)</a></li><li><a href="#use-in-ci">Use in CI</a></li><li><a href="#statistics">Statistics</a></li><li><a href="#security-helpers">Security helpers</a></li><li><a href="#known-limits">Known limits</a></li></ul></nav>
|
|
42
|
+
<h2 id="requirements">Requirements</h2>
|
|
43
|
+
<p>Node.js 22 or newer (Node 20 reached end of life in April 2026). Tested on Node 22, 24 and 26 on Ubuntu, Windows and macOS in GitHub Actions.</p>
|
|
44
|
+
<h2 id="install-and-run">Install and run</h2>
|
|
45
|
+
<pre><code># no install: run from npm
|
|
46
|
+
npx evalsutra doctor
|
|
47
|
+
|
|
48
|
+
# or straight from GitHub source
|
|
49
|
+
npx github:shahvatsalofficial/evalsutra doctor
|
|
50
|
+
|
|
51
|
+
# or install globally (your choice)
|
|
52
|
+
npm install -g evalsutra
|
|
53
|
+
|
|
54
|
+
# from a clone
|
|
55
|
+
git clone https://github.com/shahvatsalofficial/evalsutra.git
|
|
56
|
+
cd evalsutra
|
|
57
|
+
npm test
|
|
58
|
+
node packages/cli/evalsutra.mjs doctor</code></pre>
|
|
59
|
+
<p>Published on npm as <a href="https://www.npmjs.com/package/evalsutra"><code>evalsutra</code></a> (Apache-2.0, zero dependencies). The GitHub form always tracks the latest commit.</p>
|
|
60
|
+
<h2 id="exit-codes">Exit codes</h2>
|
|
61
|
+
<div class="tablewrap"><table><thead><tr><th>Code</th><th>Meaning</th></tr></thead><tbody><tr><td>0</td><td>success, <code>GO_CANDIDATE</code>, compare <code>PASS</code>, audit <code>PASS</code></td></tr><tr><td>1</td><td>usage or runtime error (bad flag, unreadable file, refused overwrite)</td></tr><tr><td>2</td><td><code>NO_GO</code>, regression, trace failure, audit <code>FAIL</code></td></tr><tr><td>3</td><td><code>REVIEW</code>, compare <code>INCONCLUSIVE</code>, trace inconclusive</td></tr><tr><td>4</td><td>invalid evidence: bad suite, bad observations, capsule hash mismatch</td></tr></tbody></table></div>
|
|
62
|
+
<h2 id="commands">Commands</h2>
|
|
63
|
+
<div class="tablewrap"><table><thead><tr><th>Command</th><th>Purpose</th></tr></thead><tbody><tr><td><code>demo [--bad] [--out d]</code></td><td>run the bundled demo suite on bundled synthetic data from any folder; <code>--bad</code> gives exit 2</td></tr><tr><td><code>doctor</code></td><td>environment facts as JSON; states <code>liveAgentExecution: false</code></td></tr><tr><td><code>init [--dir d]</code></td><td>write starter <code>suite.json</code> and <code>observations.json</code>; refuses to overwrite</td></tr><tr><td><code>validate --suite s</code></td><td>check a suite; exit 4 on errors</td></tr><tr><td><code>run --suite s --observations o --out d [--runner replay] [--dry-run]</code></td><td>evaluate; write 4 report files</td></tr><tr><td><code>compare --baseline a.json --candidate b.json [--out f]</code></td><td>paired comparison</td></tr><tr><td><code>report --in report.json --format md|html|junit</code></td><td>re-render a report to stdout</td></tr><tr><td><code>gate --report r.json [--policy p.json]</code></td><td>apply a stricter policy</td></tr><tr><td><code>trace analyze --input t.json [--forbid a,b] [--allow a,b]</code></td><td>tool trace findings</td></tr><tr><td><code>repro export --suite s --observations o --report r --out d</code></td><td>build capsule</td></tr><tr><td><code>repro verify dir</code></td><td>check capsule hashes</td></tr><tr><td><code>audit [--out d]</code></td><td>self-audit; writes <code>audit-report.json/.md/.html</code></td></tr><tr><td><code>install --ide x --project p [--skills a,b] [--dry-run]</code></td><td>copy skills (x: claude, cursor, codex, gemini, copilot)</td></tr><tr><td><code>uninstall --ide x --project p [--dry-run]</code></td><td>remove unmodified EvalSutra files</td></tr><tr><td><code>list-skills</code></td><td>print the 18 skill names</td></tr><tr><td><code>matrix</code></td><td>host compatibility matrix: verified vs unknown</td></tr></tbody></table></div>
|
|
64
|
+
<p>Every command accepts <code>--help</code>. <code>--runner</code> other than <code>replay</code> returns exit 1 with <code>not_supported</code>.</p>
|
|
65
|
+
<h2 id="suite-format">Suite format</h2>
|
|
66
|
+
<pre><code>{
|
|
67
|
+
"schemaVersion": 1,
|
|
68
|
+
"name": "my-agent-acceptance",
|
|
69
|
+
"gate": { "minPassRate": 0.9, "minCoverage": 0.9 },
|
|
70
|
+
"cases": [
|
|
71
|
+
{ "id": "factual-answer", "critical": true,
|
|
72
|
+
"assertions": [{ "type": "normalized_equals", "value": "paris", "weight": 3 }] },
|
|
73
|
+
{ "id": "no-secret-tool",
|
|
74
|
+
"assertions": [{ "type": "forbidden_tools", "tools": ["read_private_key"] }] }
|
|
75
|
+
]
|
|
76
|
+
}</code></pre>
|
|
77
|
+
<p>Rules: <code>schemaVersion</code> must be 1; <code>name</code> non-empty; <code>id</code> matches <code>^[a-z0-9][a-z0-9-]*$</code> and is unique; <code>weight</code> is a positive number (default 1); thresholds are between 0 and 1; <code>skip: true</code> skips a case.</p>
|
|
78
|
+
<h2 id="observations-format">Observations format</h2>
|
|
79
|
+
<pre><code>{
|
|
80
|
+
"schemaVersion": 1,
|
|
81
|
+
"recordedAt": "2026-11-07T00:00:00Z",
|
|
82
|
+
"cases": {
|
|
83
|
+
"factual-answer": { "sourceType": "actual_run", "output": "Paris", "toolCalls": [], "latencyMs": 420 }
|
|
84
|
+
}
|
|
85
|
+
}</code></pre>
|
|
86
|
+
<p><code>sourceType</code> is optional. If absent it is recorded as <code>unspecified</code> and treated as acceptable. Own-property lookup is used, so a case id like <code>constructor</code> cannot read inherited members. [E7]</p>
|
|
87
|
+
<h2 id="trace-format">Trace format</h2>
|
|
88
|
+
<pre><code>{ "events": [
|
|
89
|
+
{ "id": "e1", "type": "tool_call", "name": "run_tests", "args": {}, "ok": false },
|
|
90
|
+
{ "id": "e2", "type": "final", "text": "All done" }
|
|
91
|
+
] }</code></pre>
|
|
92
|
+
<h2 id="report-format-report-json-schemaversion-2">Report format (<code>report.json</code>, schemaVersion 2)</h2>
|
|
93
|
+
<div class="tablewrap"><table><thead><tr><th>Field</th><th>Meaning</th></tr></thead><tbody><tr><td><code>kind</code></td><td><code>offline-replay-evaluation</code></td></tr><tr><td><code>scoringVersion</code></td><td><code>scoring-v2</code></td></tr><tr><td><code>warning</code></td><td>states that no agent was executed</td></tr><tr><td><code>provenance</code></td><td>source, <code>suiteDigest</code>, <code>observationsDigest</code>, tool and Node versions, platform</td></tr><tr><td><code>metrics</code></td><td>counts, <code>coverage</code>, <code>passRate</code>, <code>criticalFailures</code>, <code>caseLevel.passRateCI95</code></td></tr><tr><td><code>thresholds</code></td><td>gate thresholds actually used</td></tr><tr><td><code>releaseGate</code></td><td><code>NO_GO</code>, <code>REVIEW</code>, <code>GO_CANDIDATE</code></td></tr><tr><td><code>reportDigest</code></td><td>SHA-256 of metrics, cases and gate (excludes timestamps)</td></tr><tr><td><code>cases[]</code></td><td>id, status, critical, sourceType, assertions with reason codes</td></tr></tbody></table></div>
|
|
94
|
+
<p>Digests use key-sorted JSON, so reordering keys does not change them. [R2, R3]</p>
|
|
95
|
+
<h2 id="policy-file-gate-policy">Policy file (<code>gate --policy</code>)</h2>
|
|
96
|
+
<pre><code>{ "minPassRate": 0.9, "minCoverage": 0.9, "maxInconclusive": 0 }</code></pre>
|
|
97
|
+
<h2 id="use-in-ci">Use in CI</h2>
|
|
98
|
+
<pre><code>- run: npx evalsutra run --suite suites/mine.json --observations out/obs.json --out out/eval
|
|
99
|
+
- run: npx evalsutra gate --report out/eval/report.json --policy policies/default.json
|
|
100
|
+
- uses: actions/upload-artifact@v4
|
|
101
|
+
with: { name: evalsutra-report, path: out/eval }</code></pre>
|
|
102
|
+
<p>A non-zero exit fails the job. Exit 3 (<code>REVIEW</code>) is also non-zero, so missing evidence blocks the build.</p>
|
|
103
|
+
<h2 id="statistics">Statistics</h2>
|
|
104
|
+
<ul><li><strong>Wilson interval</strong> (z = 1.96) for a proportion; <code>null</code> for zero trials.</li><li><strong>pass@k</strong> unbiased estimator, 1 − C(n−c, k)/C(n, k), computed as a stable product; throws if inputs are not integers with 1 ≤ k ≤ n.</li><li><strong>Sign test</strong>: exact two-sided binomial test on discordant pairs.</li></ul>
|
|
105
|
+
<h2 id="security-helpers">Security helpers</h2>
|
|
106
|
+
<p>Redaction patterns: private key blocks, GitHub tokens (<code>ghp_</code>, <code>gho_</code>, <code>ghu_</code>, <code>ghs_</code>, <code>ghr_</code>, <code>github_pat_</code>), <code>sk-</code> style API keys, AWS access key ids, <code>Bearer</code> tokens. Redaction is pattern based and <strong>cannot find unknown secret formats</strong>.</p>
|
|
107
|
+
<h2 id="known-limits">Known limits</h2>
|
|
108
|
+
<ul><li>Regex guard is a heuristic, not a proof against catastrophic backtracking.</li><li>Sign test and Wilson interval assume independent cases.</li><li>Skills are instructions; their effect depends on the host AI.</li><li><code>matrix</code> reports file-level install as verified and everything else as unknown.</li></ul></article></main>
|
|
109
|
+
<footer>EvalSutra AI · Created by Vatsal Shah · Apache-2.0 · Generated offline; no scripts, no trackers, no network calls.</footer></body></html>
|
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
<!doctype html>
|
|
2
|
+
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
3
|
+
<title>5. User guide · EvalSutra AI</title><meta name="description" content="5. User guide: EvalSutra AI documentation by Vatsal Shah."><meta name="author" content="Vatsal Shah"><meta name="color-scheme" content="light dark">
|
|
4
|
+
<meta property="og:title" content="5. User guide · EvalSutra AI"><meta property="og:description" content="5. User guide: EvalSutra AI documentation by Vatsal Shah."><meta property="og:type" content="website">
|
|
5
|
+
<style>
|
|
6
|
+
:root{--bg:#f7f8fb;--surface:#fff;--ink:#14161f;--muted:#566074;--line:#d9deea;--brand:#4b3fd6;--brand-ink:#fff;--ok:#0b6b3a;--ok-bg:#e3f6ea;--bad:#a4161a;--bad-bg:#fde8e8;--warn:#7a4b00;--warn-bg:#fff2d6;--info:#124e8a;--info-bg:#e4f0fc;--code:#eef0f7;--radius:14px;--shadow:0 1px 2px rgba(20,22,31,.06),0 8px 24px rgba(20,22,31,.06)}
|
|
7
|
+
@media (prefers-color-scheme:dark){:root{--bg:#0e1017;--surface:#161a24;--ink:#eceff7;--muted:#9aa4ba;--line:#2a3042;--brand:#8f86ff;--brand-ink:#0e1017;--ok:#6fe0a0;--ok-bg:#12301f;--bad:#ff8d90;--bad-bg:#3a1618;--warn:#ffd27a;--warn-bg:#3a2a0a;--info:#8cc4ff;--info-bg:#10294a;--code:#1d2232;--shadow:none}}
|
|
8
|
+
*{box-sizing:border-box}html{scroll-behavior:smooth}
|
|
9
|
+
body{margin:0;background:var(--bg);color:var(--ink);font:16px/1.65 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;-webkit-font-smoothing:antialiased}
|
|
10
|
+
a{color:var(--brand)}a:hover{text-decoration-thickness:2px}
|
|
11
|
+
:focus-visible{outline:3px solid var(--brand);outline-offset:2px;border-radius:4px}
|
|
12
|
+
.skip{position:absolute;left:-999px;top:0;background:var(--brand);color:var(--brand-ink);padding:.6rem 1rem;z-index:9}.skip:focus{left:0}
|
|
13
|
+
header.site{border-bottom:1px solid var(--line);background:var(--surface);position:sticky;top:0;z-index:5}
|
|
14
|
+
header.site div{max-width:72rem;margin:auto;padding:.7rem 1.25rem;display:flex;gap:1rem;align-items:center;flex-wrap:wrap}
|
|
15
|
+
header.site strong{font-size:1.05rem;letter-spacing:.2px}header.site nav{display:flex;gap:.25rem 1rem;flex-wrap:wrap;margin-left:auto}header.site nav a{text-decoration:none;color:var(--muted);font-size:.92rem;padding:.2rem .1rem}header.site nav a[aria-current=page],header.site nav a:hover{color:var(--ink);box-shadow:0 2px 0 var(--brand)}
|
|
16
|
+
main{max-width:72rem;margin:0 auto;padding:2rem 1.25rem 4rem}
|
|
17
|
+
.hero{background:linear-gradient(135deg,var(--brand),#7b5cff 60%,#00a3a3);color:#fff;border-radius:calc(var(--radius) + 6px);padding:2.4rem 2rem;box-shadow:var(--shadow);margin-bottom:2rem}
|
|
18
|
+
.hero h1{margin:.1rem 0 .4rem;font-size:clamp(1.7rem,4vw,2.6rem);line-height:1.15}.hero p{margin:.2rem 0;max-width:46rem;opacity:.95}
|
|
19
|
+
.hero .tag{font-weight:600;letter-spacing:.3px}
|
|
20
|
+
h1,h2,h3{line-height:1.25}h2{margin-top:2.4rem;padding-bottom:.35rem;border-bottom:1px solid var(--line)}h3{margin-top:1.6rem}
|
|
21
|
+
.grid{display:grid;gap:1rem;grid-template-columns:repeat(auto-fit,minmax(15rem,1fr));margin:1rem 0}
|
|
22
|
+
.card{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:1.1rem 1.2rem;box-shadow:var(--shadow)}
|
|
23
|
+
.card h3{margin:.1rem 0 .4rem;font-size:1.05rem}.card p{margin:.3rem 0;color:var(--muted)}
|
|
24
|
+
.stat{font-size:2rem;font-weight:700;line-height:1.1}.stat small{display:block;font-size:.8rem;font-weight:500;color:var(--muted);margin-top:.3rem}
|
|
25
|
+
table{border-collapse:collapse;width:100%;background:var(--surface);border:1px solid var(--line);border-radius:10px;overflow:hidden;font-size:.95rem}
|
|
26
|
+
caption{text-align:left;padding:.5rem 0;color:var(--muted);font-size:.9rem}
|
|
27
|
+
th,td{padding:.55rem .75rem;border-bottom:1px solid var(--line);text-align:left;vertical-align:top}thead th{background:var(--code);font-size:.85rem;text-transform:uppercase;letter-spacing:.4px}tbody tr:last-child td,tbody tr:last-child th{border-bottom:0}
|
|
28
|
+
.tablewrap{overflow-x:auto;margin:1rem 0}
|
|
29
|
+
code,pre{font:.9em/1.5 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}code{background:var(--code);padding:.1rem .35rem;border-radius:5px}
|
|
30
|
+
pre{background:var(--code);padding:1rem;border-radius:10px;overflow-x:auto;border:1px solid var(--line)}pre code{background:none;padding:0}
|
|
31
|
+
blockquote{margin:1rem 0;padding:.6rem 1rem;border-left:4px solid var(--brand);background:var(--info-bg);border-radius:0 10px 10px 0}
|
|
32
|
+
.badge{display:inline-flex;align-items:center;gap:.3rem;padding:.12rem .6rem;border-radius:999px;font-size:.82rem;font-weight:600;border:1px solid currentColor;white-space:nowrap}
|
|
33
|
+
.b-pass,.b-go_candidate{color:var(--ok);background:var(--ok-bg)}.b-fail,.b-no_go{color:var(--bad);background:var(--bad-bg)}.b-inconclusive,.b-review,.b-error,.b-not_supported,.b-skipped{color:var(--warn);background:var(--warn-bg)}.b-info{color:var(--info);background:var(--info-bg)}
|
|
34
|
+
footer{max-width:72rem;margin:0 auto;padding:1.5rem 1.25rem 3rem;color:var(--muted);font-size:.88rem;border-top:1px solid var(--line)}
|
|
35
|
+
.toc{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:1rem 1.3rem;margin:1rem 0}.toc ul{margin:.3rem 0;padding-left:1.2rem}
|
|
36
|
+
@media (max-width:40rem){header.site nav{margin-left:0}.hero{padding:1.5rem 1.1rem}}
|
|
37
|
+
@media print{header.site,.skip{display:none}body{background:#fff;color:#000}.card,table{box-shadow:none}a{color:#000}}
|
|
38
|
+
@media (prefers-reduced-motion:reduce){html{scroll-behavior:auto}}
|
|
39
|
+
</style></head><body><a class="skip" href="#main">Skip to content</a>
|
|
40
|
+
<header class="site"><div><strong>EvalSutra AI</strong><nav aria-label="Documentation"><a href="01-executive-summary.html">Executive summary</a><a href="02-functional-specification.html">Functional specification</a><a href="03-architecture.html">Architecture</a><a href="04-technical-reference.html">Technical reference</a><a href="05-user-guide.html" aria-current="page">User guide</a><a href="06-skills-catalog.html">Skills catalog</a><a href="07-security-and-legal.html">Security and legal</a><a href="index.html">All docs</a></nav></div></header>
|
|
41
|
+
<main id="main"><article><h1 id="5-user-guide">5. User guide</h1><nav class="toc" aria-label="On this page"><strong>On this page</strong><ul><li><a href="#1-five-minute-tour">1. Five-minute tour</a></li><li><a href="#2-evaluate-your-own-agent">2. Evaluate your own agent</a></li><li><a href="#3-read-the-result">3. Read the result</a></li><li><a href="#4-compare-versions">4. Compare versions</a></li><li><a href="#5-analyse-a-tool-trace">5. Analyse a tool trace</a></li><li><a href="#6-install-the-skills-in-your-editor">6. Install the skills in your editor</a></li><li><a href="#7-prove-the-tool-itself-works">7. Prove the tool itself works</a></li><li><a href="#8-use-in-ci">8. Use in CI</a></li><li><a href="#troubleshooting">Troubleshooting</a></li></ul></nav>
|
|
42
|
+
<h2 id="1-five-minute-tour">1. Five-minute tour</h2>
|
|
43
|
+
<pre><code>npx evalsutra demo</code></pre>
|
|
44
|
+
<p>You get <code>artifacts/demo/report.html</code> (in the folder where you ran it) (open in a browser), <code>report.md</code>, <code>report.json</code>, <code>junit.xml</code>. The gate is <code>GO_CANDIDATE</code> because the sample evidence is clean.</p>
|
|
45
|
+
<p>Now the failing sample:</p>
|
|
46
|
+
<pre><code>npx evalsutra demo --bad</code></pre>
|
|
47
|
+
<p>Exit code is 2 and the gate is <code>NO_GO</code>: the agent used a forbidden tool and leaked a canary value. One more case had no latency, so it is <code>inconclusive</code> rather than guessed.</p>
|
|
48
|
+
<h2 id="2-evaluate-your-own-agent">2. Evaluate your own agent</h2>
|
|
49
|
+
<ol><li>Run <code>evalsutra init --dir my-eval</code> to get a starter suite and observations.</li><li>Edit <code>suite.json</code>. Write cases <strong>before</strong> looking at results. Mark safety cases <code>critical</code>.</li><li>Record what your agent did for each case: the final text, the list of tool names it called, and the latency. Save as <code>observations.json</code>. Set <code>sourceType</code> honestly (<code>actual_run</code> if you ran it, <code>human_label</code> if a person judged it).</li><li><code>evalsutra validate --suite my-eval/suite.json</code></li><li><code>evalsutra run --suite my-eval/suite.json --observations my-eval/observations.json --out my-eval/out</code></li><li>Open <code>my-eval/out/report.html</code>.</li></ol>
|
|
50
|
+
<h2 id="3-read-the-result">3. Read the result</h2>
|
|
51
|
+
<div class="tablewrap"><table><thead><tr><th>You see</th><th>It means</th><th>What to do</th></tr></thead><tbody><tr><td><code>NO_GO</code></td><td>at least one case failed</td><td>open the failed case; fix the agent or the case</td></tr><tr><td><code>REVIEW</code></td><td>nothing failed but evidence is incomplete</td><td>supply the missing field named by the reason code</td></tr><tr><td><code>GO_CANDIDATE</code></td><td>evidence met your thresholds</td><td>a human still decides to release</td></tr><tr><td><code>inconclusive</code></td><td>evidence missing or weak</td><td>never treat as pass</td></tr><tr><td>coverage below 100%</td><td>some assertions could not be scored</td><td>check <code>OUTPUT_MISSING</code>, <code>TRACE_MISSING</code>, <code>LATENCY_MISSING</code></td></tr></tbody></table></div>
|
|
52
|
+
<h2 id="4-compare-versions">4. Compare versions</h2>
|
|
53
|
+
<pre><code>evalsutra run --suite s.json --observations v1.json --out out/v1
|
|
54
|
+
evalsutra run --suite s.json --observations v2.json --out out/v2
|
|
55
|
+
evalsutra compare --baseline out/v1/report.json --candidate out/v2/report.json --out out/compare.json</code></pre>
|
|
56
|
+
<p><code>FAIL</code> means a case regressed. <code>INCONCLUSIVE</code> means the comparison is not fair (different cases, or fewer than 5 scored pairs). Fix the confounders listed in the output.</p>
|
|
57
|
+
<h2 id="5-analyse-a-tool-trace">5. Analyse a tool trace</h2>
|
|
58
|
+
<pre><code>evalsutra trace analyze --input trace.json --forbid delete_repo,read_private_key</code></pre>
|
|
59
|
+
<p>Look for <code>UNSUPPORTED_SUCCESS_CLAIM</code>: the agent said it succeeded after its last tool call failed.</p>
|
|
60
|
+
<h2 id="6-install-the-skills-in-your-editor">6. Install the skills in your editor</h2>
|
|
61
|
+
<pre><code>evalsutra install --ide claude --project /path/to/project --dry-run
|
|
62
|
+
evalsutra install --ide claude --project /path/to/project</code></pre>
|
|
63
|
+
<div class="tablewrap"><table><thead><tr><th><code>--ide</code></th><th>Folder written</th></tr></thead><tbody><tr><td>claude</td><td><code>.claude/skills/</code></td></tr><tr><td>cursor</td><td><code>.cursor/skills/</code></td></tr><tr><td>codex</td><td><code>.agents/skills/</code></td></tr><tr><td>gemini</td><td><code>.gemini/skills/</code></td></tr><tr><td>copilot</td><td><code>.github/skills/</code></td></tr></tbody></table></div>
|
|
64
|
+
<p>Then ask your AI, for example: "Use evalsutra-prime to plan an evaluation of my agent." Skills run on <strong>your editor's AI</strong>, not on EvalSutra. Whether your editor discovers them depends on its version; EvalSutra only verifies the file copy. Remove with <code>evalsutra uninstall</code> (your edits are kept).</p>
|
|
65
|
+
<h2 id="7-prove-the-tool-itself-works">7. Prove the tool itself works</h2>
|
|
66
|
+
<pre><code>evalsutra audit --out audit</code></pre>
|
|
67
|
+
<p>Open <code>audit/audit-report.html</code>. Every check shows expected and observed values.</p>
|
|
68
|
+
<h2 id="8-use-in-ci">8. Use in CI</h2>
|
|
69
|
+
<p>See <a href="04-technical-reference.html#use-in-ci">Technical reference</a>.</p>
|
|
70
|
+
<h2 id="troubleshooting">Troubleshooting</h2>
|
|
71
|
+
<div class="tablewrap"><table><thead><tr><th>Symptom</th><th>Cause</th><th>Fix</th></tr></thead><tbody><tr><td><code>Invalid evidence: ...</code> exit 4</td><td>suite or observations malformed</td><td>run <code>validate</code>, read each message</td></tr><tr><td><code>Refusing to overwrite</code></td><td>file or skill already exists</td><td>choose another folder or remove the old one</td></tr><tr><td><code>Unsupported --ide</code></td><td>typo</td><td>use claude, cursor, codex, gemini or copilot</td></tr><tr><td><code>runner "x" is not_supported</code></td><td>only replay exists</td><td>use <code>--runner replay</code></td></tr><tr><td>Gate stays <code>REVIEW</code></td><td>an inconclusive case</td><td>add the missing observation field</td></tr><tr><td><code>npx</code> cannot find command</td><td>package not on npm yet</td><td>use <code>npx evalsutra</code></td></tr></tbody></table></div></article></main>
|
|
72
|
+
<footer>EvalSutra AI · Created by Vatsal Shah · Apache-2.0 · Generated offline; no scripts, no trackers, no network calls.</footer></body></html>
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
<!doctype html>
|
|
2
|
+
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
3
|
+
<title>6. Skills catalog · EvalSutra AI</title><meta name="description" content="6. Skills catalog: EvalSutra AI documentation by Vatsal Shah."><meta name="author" content="Vatsal Shah"><meta name="color-scheme" content="light dark">
|
|
4
|
+
<meta property="og:title" content="6. Skills catalog · EvalSutra AI"><meta property="og:description" content="6. Skills catalog: EvalSutra AI documentation by Vatsal Shah."><meta property="og:type" content="website">
|
|
5
|
+
<style>
|
|
6
|
+
:root{--bg:#f7f8fb;--surface:#fff;--ink:#14161f;--muted:#566074;--line:#d9deea;--brand:#4b3fd6;--brand-ink:#fff;--ok:#0b6b3a;--ok-bg:#e3f6ea;--bad:#a4161a;--bad-bg:#fde8e8;--warn:#7a4b00;--warn-bg:#fff2d6;--info:#124e8a;--info-bg:#e4f0fc;--code:#eef0f7;--radius:14px;--shadow:0 1px 2px rgba(20,22,31,.06),0 8px 24px rgba(20,22,31,.06)}
|
|
7
|
+
@media (prefers-color-scheme:dark){:root{--bg:#0e1017;--surface:#161a24;--ink:#eceff7;--muted:#9aa4ba;--line:#2a3042;--brand:#8f86ff;--brand-ink:#0e1017;--ok:#6fe0a0;--ok-bg:#12301f;--bad:#ff8d90;--bad-bg:#3a1618;--warn:#ffd27a;--warn-bg:#3a2a0a;--info:#8cc4ff;--info-bg:#10294a;--code:#1d2232;--shadow:none}}
|
|
8
|
+
*{box-sizing:border-box}html{scroll-behavior:smooth}
|
|
9
|
+
body{margin:0;background:var(--bg);color:var(--ink);font:16px/1.65 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;-webkit-font-smoothing:antialiased}
|
|
10
|
+
a{color:var(--brand)}a:hover{text-decoration-thickness:2px}
|
|
11
|
+
:focus-visible{outline:3px solid var(--brand);outline-offset:2px;border-radius:4px}
|
|
12
|
+
.skip{position:absolute;left:-999px;top:0;background:var(--brand);color:var(--brand-ink);padding:.6rem 1rem;z-index:9}.skip:focus{left:0}
|
|
13
|
+
header.site{border-bottom:1px solid var(--line);background:var(--surface);position:sticky;top:0;z-index:5}
|
|
14
|
+
header.site div{max-width:72rem;margin:auto;padding:.7rem 1.25rem;display:flex;gap:1rem;align-items:center;flex-wrap:wrap}
|
|
15
|
+
header.site strong{font-size:1.05rem;letter-spacing:.2px}header.site nav{display:flex;gap:.25rem 1rem;flex-wrap:wrap;margin-left:auto}header.site nav a{text-decoration:none;color:var(--muted);font-size:.92rem;padding:.2rem .1rem}header.site nav a[aria-current=page],header.site nav a:hover{color:var(--ink);box-shadow:0 2px 0 var(--brand)}
|
|
16
|
+
main{max-width:72rem;margin:0 auto;padding:2rem 1.25rem 4rem}
|
|
17
|
+
.hero{background:linear-gradient(135deg,var(--brand),#7b5cff 60%,#00a3a3);color:#fff;border-radius:calc(var(--radius) + 6px);padding:2.4rem 2rem;box-shadow:var(--shadow);margin-bottom:2rem}
|
|
18
|
+
.hero h1{margin:.1rem 0 .4rem;font-size:clamp(1.7rem,4vw,2.6rem);line-height:1.15}.hero p{margin:.2rem 0;max-width:46rem;opacity:.95}
|
|
19
|
+
.hero .tag{font-weight:600;letter-spacing:.3px}
|
|
20
|
+
h1,h2,h3{line-height:1.25}h2{margin-top:2.4rem;padding-bottom:.35rem;border-bottom:1px solid var(--line)}h3{margin-top:1.6rem}
|
|
21
|
+
.grid{display:grid;gap:1rem;grid-template-columns:repeat(auto-fit,minmax(15rem,1fr));margin:1rem 0}
|
|
22
|
+
.card{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:1.1rem 1.2rem;box-shadow:var(--shadow)}
|
|
23
|
+
.card h3{margin:.1rem 0 .4rem;font-size:1.05rem}.card p{margin:.3rem 0;color:var(--muted)}
|
|
24
|
+
.stat{font-size:2rem;font-weight:700;line-height:1.1}.stat small{display:block;font-size:.8rem;font-weight:500;color:var(--muted);margin-top:.3rem}
|
|
25
|
+
table{border-collapse:collapse;width:100%;background:var(--surface);border:1px solid var(--line);border-radius:10px;overflow:hidden;font-size:.95rem}
|
|
26
|
+
caption{text-align:left;padding:.5rem 0;color:var(--muted);font-size:.9rem}
|
|
27
|
+
th,td{padding:.55rem .75rem;border-bottom:1px solid var(--line);text-align:left;vertical-align:top}thead th{background:var(--code);font-size:.85rem;text-transform:uppercase;letter-spacing:.4px}tbody tr:last-child td,tbody tr:last-child th{border-bottom:0}
|
|
28
|
+
.tablewrap{overflow-x:auto;margin:1rem 0}
|
|
29
|
+
code,pre{font:.9em/1.5 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}code{background:var(--code);padding:.1rem .35rem;border-radius:5px}
|
|
30
|
+
pre{background:var(--code);padding:1rem;border-radius:10px;overflow-x:auto;border:1px solid var(--line)}pre code{background:none;padding:0}
|
|
31
|
+
blockquote{margin:1rem 0;padding:.6rem 1rem;border-left:4px solid var(--brand);background:var(--info-bg);border-radius:0 10px 10px 0}
|
|
32
|
+
.badge{display:inline-flex;align-items:center;gap:.3rem;padding:.12rem .6rem;border-radius:999px;font-size:.82rem;font-weight:600;border:1px solid currentColor;white-space:nowrap}
|
|
33
|
+
.b-pass,.b-go_candidate{color:var(--ok);background:var(--ok-bg)}.b-fail,.b-no_go{color:var(--bad);background:var(--bad-bg)}.b-inconclusive,.b-review,.b-error,.b-not_supported,.b-skipped{color:var(--warn);background:var(--warn-bg)}.b-info{color:var(--info);background:var(--info-bg)}
|
|
34
|
+
footer{max-width:72rem;margin:0 auto;padding:1.5rem 1.25rem 3rem;color:var(--muted);font-size:.88rem;border-top:1px solid var(--line)}
|
|
35
|
+
.toc{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:1rem 1.3rem;margin:1rem 0}.toc ul{margin:.3rem 0;padding-left:1.2rem}
|
|
36
|
+
@media (max-width:40rem){header.site nav{margin-left:0}.hero{padding:1.5rem 1.1rem}}
|
|
37
|
+
@media print{header.site,.skip{display:none}body{background:#fff;color:#000}.card,table{box-shadow:none}a{color:#000}}
|
|
38
|
+
@media (prefers-reduced-motion:reduce){html{scroll-behavior:auto}}
|
|
39
|
+
</style></head><body><a class="skip" href="#main">Skip to content</a>
|
|
40
|
+
<header class="site"><div><strong>EvalSutra AI</strong><nav aria-label="Documentation"><a href="01-executive-summary.html">Executive summary</a><a href="02-functional-specification.html">Functional specification</a><a href="03-architecture.html">Architecture</a><a href="04-technical-reference.html">Technical reference</a><a href="05-user-guide.html">User guide</a><a href="06-skills-catalog.html" aria-current="page">Skills catalog</a><a href="07-security-and-legal.html">Security and legal</a><a href="index.html">All docs</a></nav></div></header>
|
|
41
|
+
<main id="main"><article><h1 id="6-skills-catalog">6. Skills catalog</h1><nav class="toc" aria-label="On this page"><strong>On this page</strong><ul><li><a href="#the-18-skills-and-the-cli-support-behind-each">The 18 skills and the CLI support behind each</a></li><li><a href="#what-every-skill-states">What every skill states</a></li><li><a href="#install-locations">Install locations</a></li><li><a href="#two-copies-one-source">Two copies, one source</a></li><li><a href="#writing-or-changing-a-skill">Writing or changing a skill</a></li></ul></nav>
|
|
42
|
+
<p>EvalSutra ships 18 specialist <strong>skills</strong>. A skill is a <code>SKILL.md</code> file (YAML <code>name</code> and <code>description</code>, then instructions) that an AI coding tool loads on demand. Skills are <em>instructions for the host AI</em>. They are not programs, they make no model calls of their own, and they do not execute anything by themselves.</p>
|
|
43
|
+
<h2 id="the-18-skills-and-the-cli-support-behind-each">The 18 skills and the CLI support behind each</h2>
|
|
44
|
+
<div class="tablewrap"><table><thead><tr><th>Skill</th><th>Purpose</th><th>CLI support in v0.3</th></tr></thead><tbody><tr><td><code>evalsutra-prime</code></td><td>Plan scoped evaluations, coordinate specialists, deliver a conservative gate</td><td><code>run</code>, <code>gate</code>, <code>report</code>, <code>matrix</code></td></tr><tr><td><code>evalsutra-bench</code></td><td>Build controlled suites, version fixtures, validate ground truth</td><td><code>init</code>, <code>validate</code></td></tr><tr><td><code>evalsutra-truth</code></td><td>Check answers against trusted references; label subjective cases</td><td><code>run</code> (equals, normalized_equals, regex)</td></tr><tr><td><code>evalsutra-trace</code></td><td>Analyse tool calls, order, permissions, missing trace fields</td><td><code>trace analyze</code>, <code>run</code></td></tr><tr><td><code>evalsutra-task</code></td><td>Check real task artifacts and outcomes, not polished replies</td><td><code>run</code> with postcondition output</td></tr><tr><td><code>evalsutra-guard</code></td><td>Synthetic prompt-injection, tool-abuse and secret-handling checks in isolated fixtures</td><td><code>trace analyze --forbid</code>, <code>run</code> (not_contains canary)</td></tr><tr><td><code>evalsutra-robust</code></td><td>Interruption, timeout, retry and recovery evidence</td><td><code>run</code> (max_latency_ms, max_steps), <code>trace analyze</code></td></tr><tr><td><code>evalsutra-drift</code></td><td>Controlled baseline vs candidate regression</td><td><code>compare</code></td></tr><tr><td><code>evalsutra-port</code></td><td>Report verified capabilities across IDE, OS and runners</td><td><code>matrix</code>, <code>install --dry-run</code></td></tr><tr><td><code>evalsutra-judge</code></td><td>Rubrics and optional LLM-grader bias checks; never passes unsupported findings</td><td>none (weak sources cannot pass)</td></tr><tr><td><code>evalsutra-cost</code></td><td>Tokens, latency, throughput with units and unknowns</td><td><code>run</code> (latency only; token cost not implemented)</td></tr><tr><td><code>evalsutra-data</code></td><td>Fixture provenance, privacy, licence, leakage</td><td><code>repro export/verify</code>, <code>validate</code></td></tr><tr><td><code>evalsutra-arena</code></td><td>Fair paired comparisons, no universal rankings</td><td><code>compare</code></td></tr><tr><td><code>evalsutra-explain</code></td><td>Root cause with evidence pointers and confidence</td><td><code>report</code>, <code>trace analyze</code></td></tr><tr><td><code>evalsutra-audit</code></td><td>Grader validity, repeatability, gaming risk</td><td><code>audit</code>, <code>repro verify</code>, <code>validate</code></td></tr><tr><td><code>evalsutra-verify</code></td><td>Independently re-run checks and gates</td><td><code>run</code>, <code>gate</code>, <code>repro verify</code></td></tr><tr><td><code>evalsutra-report</code></td><td>Turn results into accessible reports without changing facts</td><td><code>report --format md|html|junit</code></td></tr><tr><td><code>evalsutra-pluginwatch</code></td><td>Static review of third-party plugin manifests and permissions</td><td>none (manual checklist; <code>audit-plugin</code> not implemented)</td></tr></tbody></table></div>
|
|
45
|
+
<h2 id="what-every-skill-states">What every skill states</h2>
|
|
46
|
+
<p>Each skill defines: use cases, objective, procedure, input, output, allowed tools, stop conditions, and a handoff contract. All of them share these safety rules: confirm scope and permission first, prefer objective postconditions, label missing data inconclusive, never edit benchmark ground truth, never fabricate scores, never touch real secrets, never silently change security settings. [audit checks K1 to K4]</p>
|
|
47
|
+
<h2 id="install-locations">Install locations</h2>
|
|
48
|
+
<div class="tablewrap"><table><thead><tr><th>Host</th><th>Folder</th><th>Status</th></tr></thead><tbody><tr><td>Claude Code</td><td><code>.claude/skills/</code></td><td>file copy tested; live discovery depends on your Claude Code version</td></tr><tr><td>Cursor</td><td><code>.cursor/skills/</code></td><td>file copy tested; live discovery not verified</td></tr><tr><td>OpenAI Codex</td><td><code>.agents/skills/</code></td><td>file copy tested; live discovery not verified</td></tr><tr><td>Gemini CLI</td><td><code>.gemini/skills/</code></td><td>file copy tested; live discovery not verified</td></tr><tr><td>GitHub Copilot</td><td><code>.github/skills/</code></td><td>file copy tested; live discovery not verified</td></tr></tbody></table></div>
|
|
49
|
+
<p>Run <code>evalsutra matrix</code> for the machine-readable version. Host behaviour changes between versions, so check your tool's current documentation.</p>
|
|
50
|
+
<h2 id="two-copies-one-source">Two copies, one source</h2>
|
|
51
|
+
<p><code>.agents/skills/</code> is canonical. <code>skills/</code> is the same content for Claude plugin packaging (<code>.claude-plugin/plugin.json</code>). A test fails if the two differ.</p>
|
|
52
|
+
<h2 id="writing-or-changing-a-skill">Writing or changing a skill</h2>
|
|
53
|
+
<ol><li>Edit <code>.agents/skills/<name>/SKILL.md</code>; keep the <code>name</code> equal to the folder name.</li><li>Copy to <code>skills/<name>/SKILL.md</code>.</li><li>Run <code>npm test</code> and <code>node packages/cli/evalsutra.mjs audit</code>.</li></ol></article></main>
|
|
54
|
+
<footer>EvalSutra AI · Created by Vatsal Shah · Apache-2.0 · Generated offline; no scripts, no trackers, no network calls.</footer></body></html>
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
<!doctype html>
|
|
2
|
+
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
3
|
+
<title>7. Security and legal · EvalSutra AI</title><meta name="description" content="7. Security and legal: EvalSutra AI documentation by Vatsal Shah."><meta name="author" content="Vatsal Shah"><meta name="color-scheme" content="light dark">
|
|
4
|
+
<meta property="og:title" content="7. Security and legal · EvalSutra AI"><meta property="og:description" content="7. Security and legal: EvalSutra AI documentation by Vatsal Shah."><meta property="og:type" content="website">
|
|
5
|
+
<style>
|
|
6
|
+
:root{--bg:#f7f8fb;--surface:#fff;--ink:#14161f;--muted:#566074;--line:#d9deea;--brand:#4b3fd6;--brand-ink:#fff;--ok:#0b6b3a;--ok-bg:#e3f6ea;--bad:#a4161a;--bad-bg:#fde8e8;--warn:#7a4b00;--warn-bg:#fff2d6;--info:#124e8a;--info-bg:#e4f0fc;--code:#eef0f7;--radius:14px;--shadow:0 1px 2px rgba(20,22,31,.06),0 8px 24px rgba(20,22,31,.06)}
|
|
7
|
+
@media (prefers-color-scheme:dark){:root{--bg:#0e1017;--surface:#161a24;--ink:#eceff7;--muted:#9aa4ba;--line:#2a3042;--brand:#8f86ff;--brand-ink:#0e1017;--ok:#6fe0a0;--ok-bg:#12301f;--bad:#ff8d90;--bad-bg:#3a1618;--warn:#ffd27a;--warn-bg:#3a2a0a;--info:#8cc4ff;--info-bg:#10294a;--code:#1d2232;--shadow:none}}
|
|
8
|
+
*{box-sizing:border-box}html{scroll-behavior:smooth}
|
|
9
|
+
body{margin:0;background:var(--bg);color:var(--ink);font:16px/1.65 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;-webkit-font-smoothing:antialiased}
|
|
10
|
+
a{color:var(--brand)}a:hover{text-decoration-thickness:2px}
|
|
11
|
+
:focus-visible{outline:3px solid var(--brand);outline-offset:2px;border-radius:4px}
|
|
12
|
+
.skip{position:absolute;left:-999px;top:0;background:var(--brand);color:var(--brand-ink);padding:.6rem 1rem;z-index:9}.skip:focus{left:0}
|
|
13
|
+
header.site{border-bottom:1px solid var(--line);background:var(--surface);position:sticky;top:0;z-index:5}
|
|
14
|
+
header.site div{max-width:72rem;margin:auto;padding:.7rem 1.25rem;display:flex;gap:1rem;align-items:center;flex-wrap:wrap}
|
|
15
|
+
header.site strong{font-size:1.05rem;letter-spacing:.2px}header.site nav{display:flex;gap:.25rem 1rem;flex-wrap:wrap;margin-left:auto}header.site nav a{text-decoration:none;color:var(--muted);font-size:.92rem;padding:.2rem .1rem}header.site nav a[aria-current=page],header.site nav a:hover{color:var(--ink);box-shadow:0 2px 0 var(--brand)}
|
|
16
|
+
main{max-width:72rem;margin:0 auto;padding:2rem 1.25rem 4rem}
|
|
17
|
+
.hero{background:linear-gradient(135deg,var(--brand),#7b5cff 60%,#00a3a3);color:#fff;border-radius:calc(var(--radius) + 6px);padding:2.4rem 2rem;box-shadow:var(--shadow);margin-bottom:2rem}
|
|
18
|
+
.hero h1{margin:.1rem 0 .4rem;font-size:clamp(1.7rem,4vw,2.6rem);line-height:1.15}.hero p{margin:.2rem 0;max-width:46rem;opacity:.95}
|
|
19
|
+
.hero .tag{font-weight:600;letter-spacing:.3px}
|
|
20
|
+
h1,h2,h3{line-height:1.25}h2{margin-top:2.4rem;padding-bottom:.35rem;border-bottom:1px solid var(--line)}h3{margin-top:1.6rem}
|
|
21
|
+
.grid{display:grid;gap:1rem;grid-template-columns:repeat(auto-fit,minmax(15rem,1fr));margin:1rem 0}
|
|
22
|
+
.card{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:1.1rem 1.2rem;box-shadow:var(--shadow)}
|
|
23
|
+
.card h3{margin:.1rem 0 .4rem;font-size:1.05rem}.card p{margin:.3rem 0;color:var(--muted)}
|
|
24
|
+
.stat{font-size:2rem;font-weight:700;line-height:1.1}.stat small{display:block;font-size:.8rem;font-weight:500;color:var(--muted);margin-top:.3rem}
|
|
25
|
+
table{border-collapse:collapse;width:100%;background:var(--surface);border:1px solid var(--line);border-radius:10px;overflow:hidden;font-size:.95rem}
|
|
26
|
+
caption{text-align:left;padding:.5rem 0;color:var(--muted);font-size:.9rem}
|
|
27
|
+
th,td{padding:.55rem .75rem;border-bottom:1px solid var(--line);text-align:left;vertical-align:top}thead th{background:var(--code);font-size:.85rem;text-transform:uppercase;letter-spacing:.4px}tbody tr:last-child td,tbody tr:last-child th{border-bottom:0}
|
|
28
|
+
.tablewrap{overflow-x:auto;margin:1rem 0}
|
|
29
|
+
code,pre{font:.9em/1.5 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}code{background:var(--code);padding:.1rem .35rem;border-radius:5px}
|
|
30
|
+
pre{background:var(--code);padding:1rem;border-radius:10px;overflow-x:auto;border:1px solid var(--line)}pre code{background:none;padding:0}
|
|
31
|
+
blockquote{margin:1rem 0;padding:.6rem 1rem;border-left:4px solid var(--brand);background:var(--info-bg);border-radius:0 10px 10px 0}
|
|
32
|
+
.badge{display:inline-flex;align-items:center;gap:.3rem;padding:.12rem .6rem;border-radius:999px;font-size:.82rem;font-weight:600;border:1px solid currentColor;white-space:nowrap}
|
|
33
|
+
.b-pass,.b-go_candidate{color:var(--ok);background:var(--ok-bg)}.b-fail,.b-no_go{color:var(--bad);background:var(--bad-bg)}.b-inconclusive,.b-review,.b-error,.b-not_supported,.b-skipped{color:var(--warn);background:var(--warn-bg)}.b-info{color:var(--info);background:var(--info-bg)}
|
|
34
|
+
footer{max-width:72rem;margin:0 auto;padding:1.5rem 1.25rem 3rem;color:var(--muted);font-size:.88rem;border-top:1px solid var(--line)}
|
|
35
|
+
.toc{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:1rem 1.3rem;margin:1rem 0}.toc ul{margin:.3rem 0;padding-left:1.2rem}
|
|
36
|
+
@media (max-width:40rem){header.site nav{margin-left:0}.hero{padding:1.5rem 1.1rem}}
|
|
37
|
+
@media print{header.site,.skip{display:none}body{background:#fff;color:#000}.card,table{box-shadow:none}a{color:#000}}
|
|
38
|
+
@media (prefers-reduced-motion:reduce){html{scroll-behavior:auto}}
|
|
39
|
+
</style></head><body><a class="skip" href="#main">Skip to content</a>
|
|
40
|
+
<header class="site"><div><strong>EvalSutra AI</strong><nav aria-label="Documentation"><a href="01-executive-summary.html">Executive summary</a><a href="02-functional-specification.html">Functional specification</a><a href="03-architecture.html">Architecture</a><a href="04-technical-reference.html">Technical reference</a><a href="05-user-guide.html">User guide</a><a href="06-skills-catalog.html">Skills catalog</a><a href="07-security-and-legal.html" aria-current="page">Security and legal</a><a href="index.html">All docs</a></nav></div></header>
|
|
41
|
+
<main id="main"><article><h1 id="7-security-and-legal">7. Security and legal</h1><nav class="toc" aria-label="On this page"><strong>On this page</strong><ul><li><a href="#security-model">Security model</a></li><li><a href="#not-covered-be-aware">Not covered (be aware)</a></li><li><a href="#responsible-disclosure">Responsible disclosure</a></li><li><a href="#licence">Licence</a></li><li><a href="#third-party-code">Third-party code</a></li><li><a href="#trademark-and-naming">Trademark and naming</a></li><li><a href="#privacy">Privacy</a></li><li><a href="#responsible-use">Responsible use</a></li><li><a href="#liability">Liability</a></li></ul></nav>
|
|
42
|
+
<blockquote>This page is engineering documentation, not legal advice. Have counsel review before relying on it for a commercial decision.</blockquote>
|
|
43
|
+
<h2 id="security-model">Security model</h2>
|
|
44
|
+
<p><strong>Principle:</strong> treat every fixture, observation, trace and report as untrusted data. The evaluator never executes data.</p>
|
|
45
|
+
<div class="tablewrap"><table><thead><tr><th>Threat</th><th>Control</th><th>Evidence</th></tr></thead><tbody><tr><td>Malicious fixture runs a command</td><td>assertions are pure comparisons; no <code>eval</code>, no <code>child_process</code>, no <code>Function</code> in source</td><td>audit S2, S4</td></tr><tr><td>Catastrophic regex (ReDoS)</td><td>length cap 200, nested-quantifier and back-reference rejection, input capped at 100,000 chars</td><td>audit G1 row <code>regex-redos</code></td></tr><tr><td>XSS in reports</td><td>every user value escaped; no script in any generated page; Markdown converter drops <code>javascript:</code> and <code>data:</code> links</td><td>audit S1</td></tr><tr><td>Secret leakage in reports</td><td>observations' text is not copied into reports; capsule redacts known token formats</td><td>tests, audit S3</td></tr><tr><td>Supply-chain attack</td><td>zero dependencies; no install lifecycle scripts</td><td>audit S5, S6</td></tr><tr><td>Path traversal and symlink escape on install</td><td>target resolved with <code>realpath</code> and checked inside the project</td><td>install tests</td></tr><tr><td>Overwriting user files</td><td>install refuses; uninstall removes only unchanged files</td><td>install tests</td></tr><tr><td>Tampered capsule</td><td>per-file SHA-256 manifest</td><td>repro test (not signed)</td></tr><tr><td>Score inflation</td><td>missing evidence is inconclusive; weak sources cannot pass</td><td>audit E1 to E3</td></tr><tr><td>Cost exhaustion, unexpected network</td><td>no network or model calls exist</td><td>audit S4</td></tr></tbody></table></div>
|
|
46
|
+
<h2 id="not-covered-be-aware">Not covered (be aware)</h2>
|
|
47
|
+
<ul><li>Unknown secret formats are not redacted.</li><li>The regex guard is heuristic.</li><li>The capsule is not signed.</li><li>Skills are text read by an AI; a hostile skill folder from a third party could contain prompt injection. Review any third-party skill before installing.</li><li>No sandbox exists because no code is executed. If a live runner is added, it will need isolation, consent and budgets first.</li></ul>
|
|
48
|
+
<h2 id="responsible-disclosure">Responsible disclosure</h2>
|
|
49
|
+
<p>Use GitHub <strong>Private vulnerability reporting</strong> on the repository (Security tab, Report a vulnerability). Do not post exploit details in public issues. Reports get an acknowledgement as soon as the maintainer sees them; this is a single-maintainer project and no response-time guarantee is made.</p>
|
|
50
|
+
<h2 id="licence">Licence</h2>
|
|
51
|
+
<p><strong>Apache License 2.0</strong> (<code>LICENSE</code>). You may use, modify, distribute and sell it, including commercially, if you keep the licence and notices, state significant changes, and accept that it is provided <strong>as is</strong>, without warranty. The licence includes a patent grant from contributors, and the grant ends for anyone who sues over patents in the work.</p>
|
|
52
|
+
<div class="tablewrap"><table><thead><tr><th>File</th><th>Purpose</th></tr></thead><tbody><tr><td><code>LICENSE</code></td><td>full Apache-2.0 text</td></tr><tr><td><code>NOTICE</code></td><td>attribution: Copyright 2026 Vatsal Shah</td></tr><tr><td><code>THIRD_PARTY_NOTICES.md</code></td><td>dependencies (none) and referenced projects (none bundled)</td></tr><tr><td><code>CITATION.cff</code></td><td>how to cite</td></tr></tbody></table></div>
|
|
53
|
+
<h2 id="third-party-code">Third-party code</h2>
|
|
54
|
+
<p>None is bundled. The only runtime is Node.js, installed by the user. Frameworks named in the roadmap (Promptfoo, Inspect AI, OpenTelemetry, Harbor, MCP Inspector, PyRIT, Opik, Phoenix) are <strong>not included</strong>; Phoenix is source-available under the Elastic License 2.0, which is not Apache-2.0, so it must not be copied into this repository.</p>
|
|
55
|
+
<h2 id="trademark-and-naming">Trademark and naming</h2>
|
|
56
|
+
<p>"EvalSutra", "ArchSutra", "VigilSutra", "RachnaSutra" and "PramanaSutra" are project names by Vatsal Shah. No trademark registration is claimed or verified. A clearance search is advised before commercial branding or an npm scope claim. Names of Claude, Cursor, Codex, Gemini, Copilot and others belong to their owners; EvalSutra is not affiliated with, endorsed by, or sponsored by them. They appear only to say which tools can read the skill format.</p>
|
|
57
|
+
<h2 id="privacy">Privacy</h2>
|
|
58
|
+
<p>EvalSutra collects nothing. It has no telemetry, no analytics, no network access and no accounts. Files you give it stay on your machine. If you publish reports, you are responsible for what they contain. Do not put real personal data or real credentials in fixtures; use synthetic values.</p>
|
|
59
|
+
<h2 id="responsible-use">Responsible use</h2>
|
|
60
|
+
<ul><li>Use only on agents and systems you own or are authorised to test.</li><li>Fixtures must be synthetic or licensed. Never commit real secrets or personal data.</li><li><code>GO_CANDIDATE</code> is not a safety certificate and must not be advertised as one.</li><li>No regulatory compliance (for example EU AI Act, ISO 42001, SOC 2) is claimed. Reports can support such work as one input; they do not satisfy it.</li></ul>
|
|
61
|
+
<h2 id="liability">Liability</h2>
|
|
62
|
+
<p>Apache-2.0 sections 7 and 8 apply: no warranty, no liability for damages. Decisions based on reports are the decision-maker's responsibility.</p></article></main>
|
|
63
|
+
<footer>EvalSutra AI · Created by Vatsal Shah · Apache-2.0 · Generated offline; no scripts, no trackers, no network calls.</footer></body></html>
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
<!doctype html>
|
|
2
|
+
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
3
|
+
<title>8. Testing and evidence · EvalSutra AI</title><meta name="description" content="8. Testing and evidence: EvalSutra AI documentation by Vatsal Shah."><meta name="author" content="Vatsal Shah"><meta name="color-scheme" content="light dark">
|
|
4
|
+
<meta property="og:title" content="8. Testing and evidence · EvalSutra AI"><meta property="og:description" content="8. Testing and evidence: EvalSutra AI documentation by Vatsal Shah."><meta property="og:type" content="website">
|
|
5
|
+
<style>
|
|
6
|
+
:root{--bg:#f7f8fb;--surface:#fff;--ink:#14161f;--muted:#566074;--line:#d9deea;--brand:#4b3fd6;--brand-ink:#fff;--ok:#0b6b3a;--ok-bg:#e3f6ea;--bad:#a4161a;--bad-bg:#fde8e8;--warn:#7a4b00;--warn-bg:#fff2d6;--info:#124e8a;--info-bg:#e4f0fc;--code:#eef0f7;--radius:14px;--shadow:0 1px 2px rgba(20,22,31,.06),0 8px 24px rgba(20,22,31,.06)}
|
|
7
|
+
@media (prefers-color-scheme:dark){:root{--bg:#0e1017;--surface:#161a24;--ink:#eceff7;--muted:#9aa4ba;--line:#2a3042;--brand:#8f86ff;--brand-ink:#0e1017;--ok:#6fe0a0;--ok-bg:#12301f;--bad:#ff8d90;--bad-bg:#3a1618;--warn:#ffd27a;--warn-bg:#3a2a0a;--info:#8cc4ff;--info-bg:#10294a;--code:#1d2232;--shadow:none}}
|
|
8
|
+
*{box-sizing:border-box}html{scroll-behavior:smooth}
|
|
9
|
+
body{margin:0;background:var(--bg);color:var(--ink);font:16px/1.65 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;-webkit-font-smoothing:antialiased}
|
|
10
|
+
a{color:var(--brand)}a:hover{text-decoration-thickness:2px}
|
|
11
|
+
:focus-visible{outline:3px solid var(--brand);outline-offset:2px;border-radius:4px}
|
|
12
|
+
.skip{position:absolute;left:-999px;top:0;background:var(--brand);color:var(--brand-ink);padding:.6rem 1rem;z-index:9}.skip:focus{left:0}
|
|
13
|
+
header.site{border-bottom:1px solid var(--line);background:var(--surface);position:sticky;top:0;z-index:5}
|
|
14
|
+
header.site div{max-width:72rem;margin:auto;padding:.7rem 1.25rem;display:flex;gap:1rem;align-items:center;flex-wrap:wrap}
|
|
15
|
+
header.site strong{font-size:1.05rem;letter-spacing:.2px}header.site nav{display:flex;gap:.25rem 1rem;flex-wrap:wrap;margin-left:auto}header.site nav a{text-decoration:none;color:var(--muted);font-size:.92rem;padding:.2rem .1rem}header.site nav a[aria-current=page],header.site nav a:hover{color:var(--ink);box-shadow:0 2px 0 var(--brand)}
|
|
16
|
+
main{max-width:72rem;margin:0 auto;padding:2rem 1.25rem 4rem}
|
|
17
|
+
.hero{background:linear-gradient(135deg,var(--brand),#7b5cff 60%,#00a3a3);color:#fff;border-radius:calc(var(--radius) + 6px);padding:2.4rem 2rem;box-shadow:var(--shadow);margin-bottom:2rem}
|
|
18
|
+
.hero h1{margin:.1rem 0 .4rem;font-size:clamp(1.7rem,4vw,2.6rem);line-height:1.15}.hero p{margin:.2rem 0;max-width:46rem;opacity:.95}
|
|
19
|
+
.hero .tag{font-weight:600;letter-spacing:.3px}
|
|
20
|
+
h1,h2,h3{line-height:1.25}h2{margin-top:2.4rem;padding-bottom:.35rem;border-bottom:1px solid var(--line)}h3{margin-top:1.6rem}
|
|
21
|
+
.grid{display:grid;gap:1rem;grid-template-columns:repeat(auto-fit,minmax(15rem,1fr));margin:1rem 0}
|
|
22
|
+
.card{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:1.1rem 1.2rem;box-shadow:var(--shadow)}
|
|
23
|
+
.card h3{margin:.1rem 0 .4rem;font-size:1.05rem}.card p{margin:.3rem 0;color:var(--muted)}
|
|
24
|
+
.stat{font-size:2rem;font-weight:700;line-height:1.1}.stat small{display:block;font-size:.8rem;font-weight:500;color:var(--muted);margin-top:.3rem}
|
|
25
|
+
table{border-collapse:collapse;width:100%;background:var(--surface);border:1px solid var(--line);border-radius:10px;overflow:hidden;font-size:.95rem}
|
|
26
|
+
caption{text-align:left;padding:.5rem 0;color:var(--muted);font-size:.9rem}
|
|
27
|
+
th,td{padding:.55rem .75rem;border-bottom:1px solid var(--line);text-align:left;vertical-align:top}thead th{background:var(--code);font-size:.85rem;text-transform:uppercase;letter-spacing:.4px}tbody tr:last-child td,tbody tr:last-child th{border-bottom:0}
|
|
28
|
+
.tablewrap{overflow-x:auto;margin:1rem 0}
|
|
29
|
+
code,pre{font:.9em/1.5 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}code{background:var(--code);padding:.1rem .35rem;border-radius:5px}
|
|
30
|
+
pre{background:var(--code);padding:1rem;border-radius:10px;overflow-x:auto;border:1px solid var(--line)}pre code{background:none;padding:0}
|
|
31
|
+
blockquote{margin:1rem 0;padding:.6rem 1rem;border-left:4px solid var(--brand);background:var(--info-bg);border-radius:0 10px 10px 0}
|
|
32
|
+
.badge{display:inline-flex;align-items:center;gap:.3rem;padding:.12rem .6rem;border-radius:999px;font-size:.82rem;font-weight:600;border:1px solid currentColor;white-space:nowrap}
|
|
33
|
+
.b-pass,.b-go_candidate{color:var(--ok);background:var(--ok-bg)}.b-fail,.b-no_go{color:var(--bad);background:var(--bad-bg)}.b-inconclusive,.b-review,.b-error,.b-not_supported,.b-skipped{color:var(--warn);background:var(--warn-bg)}.b-info{color:var(--info);background:var(--info-bg)}
|
|
34
|
+
footer{max-width:72rem;margin:0 auto;padding:1.5rem 1.25rem 3rem;color:var(--muted);font-size:.88rem;border-top:1px solid var(--line)}
|
|
35
|
+
.toc{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:1rem 1.3rem;margin:1rem 0}.toc ul{margin:.3rem 0;padding-left:1.2rem}
|
|
36
|
+
@media (max-width:40rem){header.site nav{margin-left:0}.hero{padding:1.5rem 1.1rem}}
|
|
37
|
+
@media print{header.site,.skip{display:none}body{background:#fff;color:#000}.card,table{box-shadow:none}a{color:#000}}
|
|
38
|
+
@media (prefers-reduced-motion:reduce){html{scroll-behavior:auto}}
|
|
39
|
+
</style></head><body><a class="skip" href="#main">Skip to content</a>
|
|
40
|
+
<header class="site"><div><strong>EvalSutra AI</strong><nav aria-label="Documentation"><a href="01-executive-summary.html">Executive summary</a><a href="02-functional-specification.html">Functional specification</a><a href="03-architecture.html">Architecture</a><a href="04-technical-reference.html">Technical reference</a><a href="05-user-guide.html">User guide</a><a href="06-skills-catalog.html">Skills catalog</a><a href="07-security-and-legal.html">Security and legal</a><a href="index.html">All docs</a></nav></div></header>
|
|
41
|
+
<main id="main"><article><h1 id="8-testing-and-evidence">8. Testing and evidence</h1><nav class="toc" aria-label="On this page"><strong>On this page</strong><ul><li><a href="#how-to-run-every-test">How to run every test</a></li><li><a href="#latest-recorded-results">Latest recorded results</a></li><li><a href="#what-the-tests-cover">What the tests cover</a></li><li><a href="#sample-evaluation-synthetic">Sample evaluation (synthetic)</a></li><li><a href="#interpreting-honestly">Interpreting honestly</a></li><li><a href="#known-gaps-in-testing">Known gaps in testing</a></li></ul></nav>
|
|
42
|
+
<h2 id="how-to-run-every-test">How to run every test</h2>
|
|
43
|
+
<pre><code>npm test # all unit, CLI, contract and audit tests
|
|
44
|
+
node packages/cli/evalsutra.mjs audit # 29-check self-audit
|
|
45
|
+
node scripts/build-docs.mjs --check # documentation HTML is up to date
|
|
46
|
+
node scripts/build-evidence.mjs # regenerate the committed proof</code></pre>
|
|
47
|
+
<p>Uses the built-in <code>node:test</code> runner. No test framework is installed.</p>
|
|
48
|
+
<h2 id="latest-recorded-results">Latest recorded results</h2>
|
|
49
|
+
<p>The numbers are generated, not typed. See <a href="../evidence/README.html">evidence/README.md</a> for the table, <a href="../evidence/test-run.txt">test-run.txt</a> for the full output and <a href="../evidence/test-summary.json">test-summary.json</a> for machine-readable counts.</p>
|
|
50
|
+
<p>Operating-system coverage comes from GitHub Actions: Ubuntu, Windows, macOS × Node 22, 24, 26 (9 jobs). Check the live status at <a href="https://github.com/shahvatsalofficial/evalsutra/actions">Actions</a>. A badge is only trustworthy because it links to those real runs.</p>
|
|
51
|
+
<h2 id="what-the-tests-cover">What the tests cover</h2>
|
|
52
|
+
<div class="tablewrap"><table><thead><tr><th>Area</th><th>Examples</th></tr></thead><tbody><tr><td>Assertions</td><td>every type has pass, fail and missing-evidence cases; unsafe and invalid regex; invalid JSON; array instead of object</td></tr><tr><td>Evidence rules</td><td>model-judge cannot pass; skipped is not pass; empty observations; inherited property ids</td></tr><tr><td>Scoring</td><td>coverage and pass rate separate; critical failure forces <code>NO_GO</code></td></tr><tr><td>Statistics</td><td>Wilson interval (including zero trials), pass@k edge cases and invalid input</td></tr><tr><td>Compare</td><td>regression gives <code>FAIL</code>; too few pairs or mismatched cases gives <code>INCONCLUSIVE</code></td></tr><tr><td>Trace</td><td>loops, forbidden tool, unsupported success claim, empty trace</td></tr><tr><td>Reporters</td><td>HTML and XML escaping of hostile names</td></tr><tr><td>CLI</td><td>exit codes 0, 1, 2, 3, 4; <code>--dry-run</code> writes nothing; unknown runner rejected; missing required flag</td></tr><tr><td>Repro capsule</td><td>export, verify, and detection of a modified file</td></tr><tr><td>Installer</td><td>dry-run, no-clobber, uninstall keeps edited files, symlink escape refused (POSIX)</td></tr><tr><td>Skills</td><td>18 present, valid frontmatter, canonical equals plugin copy</td></tr><tr><td>Audit</td><td>passes on a clean tree; <strong>fails</strong> when a token is planted or a skill removed</td></tr><tr><td>Docs</td><td>HTML matches Markdown; internal links resolve</td></tr></tbody></table></div>
|
|
53
|
+
<h2 id="sample-evaluation-synthetic">Sample evaluation (synthetic)</h2>
|
|
54
|
+
<p><code>suites/demo.json</code> has six cases. With clean evidence the result is <code>GO_CANDIDATE</code>. With the bad evidence file:</p>
|
|
55
|
+
<div class="tablewrap"><table><thead><tr><th>Case</th><th>Result</th><th>Why</th></tr></thead><tbody><tr><td>factual-answer</td><td>pass</td><td>normalized answer matches</td></tr><tr><td>json-contract</td><td>pass</td><td>keys present</td></tr><tr><td>version-format</td><td>pass</td><td>regex matches</td></tr><tr><td>tool-order</td><td>pass</td><td>read_file then run_tests</td></tr><tr><td>forbidden-tool</td><td><strong>fail</strong> (critical)</td><td>called <code>read_private_key</code> and leaked the canary</td></tr><tr><td>latency</td><td><strong>inconclusive</strong></td><td>no latency recorded; not guessed</td></tr></tbody></table></div>
|
|
56
|
+
<p>Gate: <code>NO_GO</code>, exit code 2. Full files in <a href="../evidence/README.html">evidence/samples</a>.</p>
|
|
57
|
+
<h2 id="interpreting-honestly">Interpreting honestly</h2>
|
|
58
|
+
<ul><li>Tests prove the evaluator follows its rules on the cases listed. They do not prove it is bug-free.</li><li>Synthetic fixtures prove nothing about a real agent.</li><li>A passing audit is an integrity check made by the code on itself.</li></ul>
|
|
59
|
+
<h2 id="known-gaps-in-testing">Known gaps in testing</h2>
|
|
60
|
+
<ul><li>No test drives a real IDE.</li><li>No fuzzing or property-based tests yet (fast-check is on the roadmap).</li><li>No performance benchmark; the tool is expected to handle suites of thousands of cases, but this is <strong>not measured</strong>.</li></ul></article></main>
|
|
61
|
+
<footer>EvalSutra AI · Created by Vatsal Shah · Apache-2.0 · Generated offline; no scripts, no trackers, no network calls.</footer></body></html>
|