evalsutra 0.0.0-stage → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/evalsutra-arena/SKILL.md +27 -0
- package/.agents/skills/evalsutra-audit/SKILL.md +27 -0
- package/.agents/skills/evalsutra-bench/SKILL.md +27 -0
- package/.agents/skills/evalsutra-cost/SKILL.md +27 -0
- package/.agents/skills/evalsutra-data/SKILL.md +27 -0
- package/.agents/skills/evalsutra-drift/SKILL.md +27 -0
- package/.agents/skills/evalsutra-explain/SKILL.md +27 -0
- package/.agents/skills/evalsutra-guard/SKILL.md +27 -0
- package/.agents/skills/evalsutra-judge/SKILL.md +27 -0
- package/.agents/skills/evalsutra-pluginwatch/SKILL.md +27 -0
- package/.agents/skills/evalsutra-port/SKILL.md +27 -0
- package/.agents/skills/evalsutra-prime/SKILL.md +27 -0
- package/.agents/skills/evalsutra-report/SKILL.md +27 -0
- package/.agents/skills/evalsutra-robust/SKILL.md +27 -0
- package/.agents/skills/evalsutra-task/SKILL.md +27 -0
- package/.agents/skills/evalsutra-trace/SKILL.md +27 -0
- package/.agents/skills/evalsutra-truth/SKILL.md +27 -0
- package/.agents/skills/evalsutra-verify/SKILL.md +27 -0
- package/.claude-plugin/plugin.json +8 -0
- package/CHANGELOG.md +33 -0
- package/CITATION.cff +14 -0
- package/CODE_OF_CONDUCT.md +2 -0
- package/CONTRIBUTING.md +2 -0
- package/LICENSE +202 -0
- package/NOTICE +5 -0
- package/README.md +161 -2
- package/SECURITY.md +17 -0
- package/SUPPORT.md +5 -0
- package/THIRD_PARTY_NOTICES.md +14 -0
- package/docs/ADVANCED_TOOLING_ROADMAP.md +244 -0
- package/docs/ARCHITECTURE.md +17 -0
- package/docs/PLUGIN_AND_SKILL_CATALOG.md +56 -0
- package/docs/STATUS.md +25 -0
- package/docs/master-claude-prompt.md +534 -0
- package/documents/evalsutra/01-executive-summary.md +52 -0
- package/documents/evalsutra/02-functional-specification.md +104 -0
- package/documents/evalsutra/03-architecture.md +78 -0
- package/documents/evalsutra/04-technical-reference.md +150 -0
- package/documents/evalsutra/05-user-guide.md +94 -0
- package/documents/evalsutra/06-skills-catalog.md +52 -0
- package/documents/evalsutra/07-security-and-legal.md +66 -0
- package/documents/evalsutra/08-testing-and-evidence.md +63 -0
- package/documents/evalsutra/09-audit-and-trust.md +54 -0
- package/documents/evalsutra/10-ci-cd-and-release.md +50 -0
- package/documents/evalsutra/11-faq-and-glossary.md +51 -0
- package/documents/evalsutra/12-contributor-handbook.md +67 -0
- package/documents/evalsutra/13-roadmap-and-status.md +48 -0
- package/documents/evalsutra/assets/architecture.svg +21 -0
- package/documents/evalsutra/evidence/README.md +39 -0
- package/documents/evalsutra/evidence/samples/audit/audit-report.html +46 -0
- package/documents/evalsutra/evidence/samples/audit/audit-report.json +317 -0
- package/documents/evalsutra/evidence/samples/audit/audit-report.md +64 -0
- package/documents/evalsutra/evidence/samples/bad/junit.xml +2 -0
- package/documents/evalsutra/evidence/samples/bad/report.html +47 -0
- package/documents/evalsutra/evidence/samples/bad/report.json +153 -0
- package/documents/evalsutra/evidence/samples/bad/report.md +22 -0
- package/documents/evalsutra/evidence/samples/compare-good-vs-bad.json +59 -0
- package/documents/evalsutra/evidence/samples/demo-suite.json +13 -0
- package/documents/evalsutra/evidence/samples/gate-bad-run.json +10 -0
- package/documents/evalsutra/evidence/samples/good/junit.xml +2 -0
- package/documents/evalsutra/evidence/samples/good/report.html +47 -0
- package/documents/evalsutra/evidence/samples/good/report.json +153 -0
- package/documents/evalsutra/evidence/samples/good/report.md +22 -0
- package/documents/evalsutra/evidence/samples/trace-analysis.json +33 -0
- package/documents/evalsutra/evidence/test-run.txt +71 -0
- package/documents/evalsutra/evidence/test-summary.json +11 -0
- package/documents/evalsutra/html/01-executive-summary.html +57 -0
- package/documents/evalsutra/html/02-functional-specification.html +75 -0
- package/documents/evalsutra/html/03-architecture.html +60 -0
- package/documents/evalsutra/html/04-technical-reference.html +109 -0
- package/documents/evalsutra/html/05-user-guide.html +72 -0
- package/documents/evalsutra/html/06-skills-catalog.html +54 -0
- package/documents/evalsutra/html/07-security-and-legal.html +63 -0
- package/documents/evalsutra/html/08-testing-and-evidence.html +61 -0
- package/documents/evalsutra/html/09-audit-and-trust.html +61 -0
- package/documents/evalsutra/html/10-ci-cd-and-release.html +61 -0
- package/documents/evalsutra/html/11-faq-and-glossary.html +57 -0
- package/documents/evalsutra/html/12-contributor-handbook.html +71 -0
- package/documents/evalsutra/html/13-roadmap-and-status.html +51 -0
- package/documents/evalsutra/html/assets/architecture.svg +21 -0
- package/documents/evalsutra/html/index.html +42 -0
- package/examples/demo-observations-bad.json +7 -0
- package/examples/demo-observations.json +7 -0
- package/examples/observations.json +10 -0
- package/examples/output/report.json +99 -0
- package/examples/output/report.md +18 -0
- package/examples/suite.json +11 -0
- package/examples/traces/bad-trace.json +5 -0
- package/llms.txt +25 -0
- package/package.json +19 -4
- package/packages/audit/audit.mjs +179 -0
- package/packages/cli/evalsutra.mjs +151 -0
- package/packages/cli/install.mjs +68 -0
- package/packages/core/capsule.mjs +31 -0
- package/packages/core/evaluate.mjs +187 -0
- package/packages/core/security.mjs +22 -0
- package/packages/core/stats.mjs +16 -0
- package/packages/core/trace.mjs +29 -0
- package/packages/reporters/index.mjs +26 -0
- package/packages/reporters/theme.mjs +53 -0
- package/policies/README.md +3 -0
- package/policies/default.json +1 -0
- package/skills/evalsutra-arena/SKILL.md +27 -0
- package/skills/evalsutra-audit/SKILL.md +27 -0
- package/skills/evalsutra-bench/SKILL.md +27 -0
- package/skills/evalsutra-cost/SKILL.md +27 -0
- package/skills/evalsutra-data/SKILL.md +27 -0
- package/skills/evalsutra-drift/SKILL.md +27 -0
- package/skills/evalsutra-explain/SKILL.md +27 -0
- package/skills/evalsutra-guard/SKILL.md +27 -0
- package/skills/evalsutra-judge/SKILL.md +27 -0
- package/skills/evalsutra-pluginwatch/SKILL.md +27 -0
- package/skills/evalsutra-port/SKILL.md +27 -0
- package/skills/evalsutra-prime/SKILL.md +27 -0
- package/skills/evalsutra-report/SKILL.md +27 -0
- package/skills/evalsutra-robust/SKILL.md +27 -0
- package/skills/evalsutra-task/SKILL.md +27 -0
- package/skills/evalsutra-trace/SKILL.md +27 -0
- package/skills/evalsutra-truth/SKILL.md +27 -0
- package/skills/evalsutra-verify/SKILL.md +27 -0
- package/suites/demo.json +13 -0
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# 9. Audit and trust
|
|
2
|
+
|
|
3
|
+
EvalSutra can audit itself with one command and show its working.
|
|
4
|
+
|
|
5
|
+
```bash
|
|
6
|
+
evalsutra audit --out audit
|
|
7
|
+
```
|
|
8
|
+
|
|
9
|
+
This writes `audit-report.md`, `audit-report.html` and `audit-report.json`. A pre-generated copy is in [evidence/samples/audit](evidence/samples/audit/audit-report.md).
|
|
10
|
+
|
|
11
|
+
## What the 29 checks cover
|
|
12
|
+
|
|
13
|
+
| Category | Checks | What is proven |
|
|
14
|
+
|---|---|---|
|
|
15
|
+
| Grader accuracy (G1 to G3) | 3 | A table of 36 known-answer cases (each assertion type, including edge cases) gives exactly the true status. Zero false positives, zero false negatives. |
|
|
16
|
+
| Evidence rules (E1 to E7) | 7 | Empty evidence never passes; model-judge evidence cannot reach `GO_CANDIDATE`; all-skipped never passes; forbidden tool plus canary leak is `NO_GO`; clean evidence is only a candidate; regressions are caught; id `constructor` reads nothing inherited |
|
|
17
|
+
| Reproducibility (R1 to R3) | 3 | Identical inputs give byte-identical results; digests ignore key order; any change changes the digest |
|
|
18
|
+
| Trace analysis (T1 to T2) | 2 | False success claims are flagged; empty trace is inconclusive |
|
|
19
|
+
| Security (S1 to S6) | 6 | Hostile suite names are escaped; shell text in fixtures is inert; secret scan over every text file; no network, process or dynamic-code APIs in source; zero dependencies; no install scripts |
|
|
20
|
+
| Skills (K1 to K4) | 4 | 18 skills; valid frontmatter; required sections present; plugin copy equals canonical |
|
|
21
|
+
| Legal (L1 to L4) | 4 | Apache-2.0 text; `package.json` licence; NOTICE and third-party notices; security, contributing, conduct, changelog files |
|
|
22
|
+
|
|
23
|
+
## Does the audit have teeth?
|
|
24
|
+
|
|
25
|
+
A check that cannot fail proves nothing. The test suite breaks the tool on purpose and requires the audit to notice: it plants a fake token in a copy of the repository and requires check S3 to fail, and it removes a skill and requires K1 to fail. See `tests/audit.test.mjs`.
|
|
26
|
+
|
|
27
|
+
## Honest limits
|
|
28
|
+
|
|
29
|
+
1. **Self-audit, not third-party attestation.** The audit runs inside the code it checks. A malicious fork could change both. For an independent view, run `npm test` and `audit` from a fresh clone of the public repository and compare with the committed evidence.
|
|
30
|
+
2. **Finite table.** Passing 36 known answers proves those behaviours, not all inputs. You can add rows to `KNOWN` in `packages/audit/audit.mjs`.
|
|
31
|
+
3. **Synthetic data.** Samples are fixtures. They show the evaluator is right about them. They say nothing about how good any real agent is.
|
|
32
|
+
4. **Operating systems.** The committed evidence was generated on one machine (see `test-summary.json`). Other operating systems are covered only by the public GitHub Actions history.
|
|
33
|
+
5. **No live-host verification.** Whether Claude Code, Cursor, Codex, Gemini CLI or Copilot discover the skills in your version is unverified.
|
|
34
|
+
|
|
35
|
+
## Independent verification recipe
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
git clone https://github.com/shahvatsalofficial/evalsutra.git
|
|
39
|
+
cd evalsutra
|
|
40
|
+
git log --oneline # see every change
|
|
41
|
+
npm test # run all tests
|
|
42
|
+
node packages/cli/evalsutra.mjs audit --out /tmp/audit
|
|
43
|
+
node scripts/build-evidence.mjs # regenerate the committed proof, then `git diff`
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
If `git diff` shows only timing and platform differences, the committed evidence is honest.
|
|
47
|
+
|
|
48
|
+
## Claims we do not make
|
|
49
|
+
|
|
50
|
+
- "Production ready" or "enterprise certified"
|
|
51
|
+
- "Works in all IDEs"
|
|
52
|
+
- "Detects all prompt injection" or "detects all secrets"
|
|
53
|
+
- "Better than" any named product
|
|
54
|
+
- Any benchmark score for a real model
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# 10. CI/CD and release
|
|
2
|
+
|
|
3
|
+
## Pipeline
|
|
4
|
+
|
|
5
|
+
Files: `.github/workflows/ci.yml` (tests) and `publish.yml` (npm release). Triggers: push and pull request for CI; a published GitHub release for npm. CI permissions: `contents: read` only. Actions are pinned to full commit SHAs (checkout v7.0.1, setup-node v7.1.0, upload-artifact v7.0.2); Dependabot proposes weekly updates.
|
|
6
|
+
|
|
7
|
+
| Job | Matrix | Steps |
|
|
8
|
+
|---|---|---|
|
|
9
|
+
| `tests` | Ubuntu, Windows, macOS × Node 22, 24, 26 | `npm test`, `npm run validate`, `npm run doctor`, `audit`, docs freshness check |
|
|
10
|
+
|
|
11
|
+
No secrets are used. No network is needed by the tests. A failing job blocks nothing automatically unless branch protection requires it; enable branch protection with the `tests` checks required.
|
|
12
|
+
|
|
13
|
+
## Lessons recorded
|
|
14
|
+
|
|
15
|
+
| Incident | Cause | Fix |
|
|
16
|
+
|---|---|---|
|
|
17
|
+
| 0.3.0 npm package: README demo failed outside a clone; audit L4 failed | demo used repo-relative paths; CONTRIBUTING.md and CODE_OF_CONDUCT.md not in `files` | added `demo` command; added files; tests assert both |
|
|
18
|
+
| Windows + Node 20 job failed with `Could not find ...tests\*.test.mjs` | PowerShell does not expand globs and Node 20 does not either | `npm test` now runs `node --test`, which discovers files itself |
|
|
19
|
+
|
|
20
|
+
## Release checklist
|
|
21
|
+
|
|
22
|
+
1. `npm test` green locally.
|
|
23
|
+
2. `node packages/cli/evalsutra.mjs audit` is `PASS`.
|
|
24
|
+
3. `node scripts/build-docs.mjs` and `node scripts/build-evidence.mjs` re-run; commit results.
|
|
25
|
+
4. Update `CHANGELOG.md`, `package.json` version, `.claude-plugin/plugin.json` version, `CITATION.cff`.
|
|
26
|
+
5. Push; wait for all CI jobs green.
|
|
27
|
+
6. Tag: `git tag vX.Y.Z && git push origin vX.Y.Z`.
|
|
28
|
+
7. `gh release create vX.Y.Z --notes-file ...`.
|
|
29
|
+
8. Optional npm publish (maintainer's account, see below).
|
|
30
|
+
|
|
31
|
+
## npm publish (maintainer action)
|
|
32
|
+
|
|
33
|
+
Published as [`evalsutra`](https://www.npmjs.com/package/evalsutra). A manual `npm publish` needs the maintainer's login and two-factor code. The repository also contains `.github/workflows/publish.yml`, which publishes with npm **trusted publishing** (OIDC, provenance, no stored token) when a GitHub release is published. **One-time maintainer setup** on npmjs.com: package `evalsutra`, Settings, Trusted Publisher, GitHub Actions, owner `shahvatsalofficial`, repository `evalsutra`, workflow `publish.yml`. Until that is configured, the workflow fails and the manual route above is the one that works. The workflow has not yet run successfully.
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
npm test && node packages/cli/evalsutra.mjs audit # must be green
|
|
37
|
+
npm pack --dry-run # inspect the file list
|
|
38
|
+
npm publish --access public
|
|
39
|
+
npx -y evalsutra@X.Y.Z doctor # verify from a clean folder
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Verify the published tarball, not just the repo: a clean-folder run of `npx evalsutra demo` and `npx evalsutra audit` found two packaging bugs in 0.3.0 (see Lessons). Prefer npm **trusted publishing** from GitHub Actions (OIDC, provenance) over a long-lived token.
|
|
43
|
+
|
|
44
|
+
## Versioning
|
|
45
|
+
|
|
46
|
+
Semantic versioning. 0.x means the CLI and report format may still change; changes are listed in `CHANGELOG.md`. Report `schemaVersion` changes are called out explicitly.
|
|
47
|
+
|
|
48
|
+
## Hardening backlog
|
|
49
|
+
|
|
50
|
+
Done: Actions pinned to SHAs, Dependabot for Actions, provenance publish workflow (untested). Still open: dependency review, secret scanning (Gitleaks), CodeQL, OpenSSF Scorecard, SBOM, signed releases. None of the open items is claimed as done.
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# 11. FAQ and glossary
|
|
2
|
+
|
|
3
|
+
## FAQ
|
|
4
|
+
|
|
5
|
+
**Does EvalSutra run my agent?** No. It grades observations you record. Live execution is planned.
|
|
6
|
+
|
|
7
|
+
**Does it use AI or send data anywhere?** No. It is deterministic code with no network access.
|
|
8
|
+
|
|
9
|
+
**Can I trust `GO_CANDIDATE`?** It means your rules were met by the evidence you supplied. It is not approval, safety proof or certification.
|
|
10
|
+
|
|
11
|
+
**Why is my result `REVIEW` instead of pass?** Some evidence is missing or weak. The reason code in the report says which field.
|
|
12
|
+
|
|
13
|
+
**Why can a model judge not pass a case?** A judge gives an opinion that can be wrong or biased. Evidence from a judge can reveal a problem but never prove success.
|
|
14
|
+
|
|
15
|
+
**Is it free for commercial use?** Yes, under Apache-2.0. Keep the licence and NOTICE.
|
|
16
|
+
|
|
17
|
+
**Which editors does it support?** Skill files install into Claude Code, Cursor, Codex, Gemini CLI and Copilot folders. Only the file copy is verified. Check each tool's current docs for skill discovery.
|
|
18
|
+
|
|
19
|
+
**Why no dependencies?** Smaller attack surface and instant `npx` start.
|
|
20
|
+
|
|
21
|
+
**How do I add my own check?** See [Contributor handbook](12-contributor-handbook.md).
|
|
22
|
+
|
|
23
|
+
**Is it on npm?** Yes: `npx evalsutra` (package `evalsutra`). The GitHub form `npx github:shahvatsalofficial/evalsutra` also works.
|
|
24
|
+
|
|
25
|
+
**How do I know the samples are not faked?** Every sample file is produced by `scripts/build-evidence.mjs` from the shipped CLI. Re-run it and diff.
|
|
26
|
+
|
|
27
|
+
**How do I report a bug or vulnerability?** Bugs: GitHub issues. Vulnerabilities: private vulnerability reporting, see [Security and legal](07-security-and-legal.md).
|
|
28
|
+
|
|
29
|
+
## Glossary
|
|
30
|
+
|
|
31
|
+
| Term | Meaning |
|
|
32
|
+
|---|---|
|
|
33
|
+
| Agent | A program that uses an AI model and tools to complete tasks |
|
|
34
|
+
| Assertion | One check on one observation |
|
|
35
|
+
| Case | A named scenario with one or more assertions |
|
|
36
|
+
| Suite | A set of cases plus gate thresholds |
|
|
37
|
+
| Observation | Recorded output, tool calls and latency for a case |
|
|
38
|
+
| Evidence source | Where an observation came from (`actual_run`, `human_label`, ...) |
|
|
39
|
+
| Inconclusive | Not enough evidence to say pass or fail |
|
|
40
|
+
| Coverage | Share of assertion weight that could be scored |
|
|
41
|
+
| Pass rate | Share of scored weight that passed |
|
|
42
|
+
| Critical case | A case whose failure forces `NO_GO` |
|
|
43
|
+
| Gate | The overall decision: `NO_GO`, `REVIEW`, `GO_CANDIDATE` |
|
|
44
|
+
| Canary | A fake secret planted to see if the agent leaks it |
|
|
45
|
+
| Wilson interval | A confidence range for a proportion that behaves well for small samples |
|
|
46
|
+
| pass@k | Probability that at least one of k attempts passes |
|
|
47
|
+
| Sign test | A test on how many cases got worse versus better |
|
|
48
|
+
| Repro capsule | A folder with redacted inputs and a hash manifest |
|
|
49
|
+
| Skill | A `SKILL.md` instruction file loaded by an AI tool |
|
|
50
|
+
| ReDoS | A regular expression that takes exponential time on crafted input |
|
|
51
|
+
| ADR | Architecture decision record |
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
# 12. Contributor handbook
|
|
2
|
+
|
|
3
|
+
## Set up
|
|
4
|
+
|
|
5
|
+
```bash
|
|
6
|
+
git clone https://github.com/shahvatsalofficial/evalsutra.git
|
|
7
|
+
cd evalsutra
|
|
8
|
+
npm test
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
Node 22+. No `npm install` is needed because there are no dependencies.
|
|
12
|
+
|
|
13
|
+
## Layout
|
|
14
|
+
|
|
15
|
+
```text
|
|
16
|
+
packages/cli command line, installer
|
|
17
|
+
packages/core evaluator, stats, security, trace, capsule
|
|
18
|
+
packages/reporters HTML, JUnit, shared design system
|
|
19
|
+
packages/audit self-audit
|
|
20
|
+
.agents/skills canonical skills (copy to skills/)
|
|
21
|
+
suites examples policies demo data and gate policy
|
|
22
|
+
tests node:test files
|
|
23
|
+
scripts build-docs, build-evidence, installers
|
|
24
|
+
documents/evalsutra this documentation (Markdown source, html/ generated)
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
## Rules of the project
|
|
28
|
+
|
|
29
|
+
1. **Evidence first.** Missing data is inconclusive. Never turn unknown into pass.
|
|
30
|
+
2. **No dependencies** without a written reason in `THIRD_PARTY_NOTICES.md`.
|
|
31
|
+
3. **No network, process or dynamic-code APIs** in `packages/`. The audit enforces this.
|
|
32
|
+
4. **Every behaviour change has a test** and, if it is a grader rule, an audit `KNOWN` row.
|
|
33
|
+
5. **Docs say only what is verified.** Planned work is labelled planned.
|
|
34
|
+
6. **Synthetic fixtures only.** No real secrets, personal data or unlicensed content.
|
|
35
|
+
|
|
36
|
+
## Add an assertion type
|
|
37
|
+
|
|
38
|
+
1. `evaluate.mjs`: add the name to the type lists; validate its fields in `validateSuite`; implement it in `evaluateAssertion`.
|
|
39
|
+
2. Add rows to `KNOWN` in `packages/audit/audit.mjs`: pass, fail and missing evidence.
|
|
40
|
+
3. Add tests in `tests/`.
|
|
41
|
+
4. Document it in [Functional specification](02-functional-specification.md) and [Technical reference](04-technical-reference.md).
|
|
42
|
+
|
|
43
|
+
## Add a skill
|
|
44
|
+
|
|
45
|
+
Create `.agents/skills/evalsutra-<name>/SKILL.md` with frontmatter and the sections Objective, Procedure, Contract, Stop conditions; copy to `skills/`. Update [Skills catalog](06-skills-catalog.md) and the count checks (K1, tests, docs).
|
|
46
|
+
|
|
47
|
+
## Rebuild generated files
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
node scripts/build-docs.mjs # HTML from Markdown
|
|
51
|
+
node scripts/build-evidence.mjs # test run, samples, audit
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
CI fails if docs HTML is stale.
|
|
55
|
+
|
|
56
|
+
## Pull request checklist
|
|
57
|
+
|
|
58
|
+
- [ ] `npm test` passes
|
|
59
|
+
- [ ] `audit` is `PASS`
|
|
60
|
+
- [ ] docs and evidence rebuilt
|
|
61
|
+
- [ ] `CHANGELOG.md` updated
|
|
62
|
+
- [ ] no secrets, no new dependencies
|
|
63
|
+
- [ ] claims in docs match what the code does
|
|
64
|
+
|
|
65
|
+
## Commit and review
|
|
66
|
+
|
|
67
|
+
Small commits, imperative subject. Maintainer review is required before release. See `CONTRIBUTING.md` and `CODE_OF_CONDUCT.md` in the repository root.
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# 13. Roadmap and status
|
|
2
|
+
|
|
3
|
+
Legend: **Done** = implemented and tested. **Planned** = not implemented. Nothing marked planned should be described to users as available.
|
|
4
|
+
|
|
5
|
+
## Status matrix (v0.4.0)
|
|
6
|
+
|
|
7
|
+
| Capability | Status |
|
|
8
|
+
|---|---|
|
|
9
|
+
| Offline replay evaluator, 11 assertion types | Done |
|
|
10
|
+
| Statuses, reason codes, evidence source rules | Done |
|
|
11
|
+
| Weighted score, coverage, Wilson interval, gate | Done |
|
|
12
|
+
| Compare with PASS / FAIL / INCONCLUSIVE and sign test | Done |
|
|
13
|
+
| Trace analysis | Done (heuristic) |
|
|
14
|
+
| Reports: JSON, Markdown, HTML, JUnit | Done |
|
|
15
|
+
| Repro capsule export and verify | Done (hash only, unsigned) |
|
|
16
|
+
| Skill installer with uninstall, dry-run, guards | Done |
|
|
17
|
+
| 18 skills | Done (instructions only) |
|
|
18
|
+
| Self-audit with MD and HTML report | Done |
|
|
19
|
+
| Documentation set with generated HTML and evidence | Done |
|
|
20
|
+
| Live agent runner | Planned |
|
|
21
|
+
| Sandbox and consented command execution | Planned |
|
|
22
|
+
| SARIF output | Planned |
|
|
23
|
+
| JSON Schema files for suite, observation, report | Planned |
|
|
24
|
+
| OpenTelemetry GenAI import and export | Planned |
|
|
25
|
+
| Promptfoo, Inspect AI, Harbor, MCP Inspector adapters | Planned |
|
|
26
|
+
| Calibrated LLM judge with human gold labels | Planned |
|
|
27
|
+
| `audit-plugin` static scanner | Planned |
|
|
28
|
+
| Property-based and fuzz tests | Planned |
|
|
29
|
+
| Server API and accessible dashboard | Planned |
|
|
30
|
+
| Verified live skill discovery in each IDE | Planned |
|
|
31
|
+
| npm publication | Done (`evalsutra` on npm); trusted publishing and provenance planned |
|
|
32
|
+
| Signed releases, SBOM, Scorecard | Planned |
|
|
33
|
+
|
|
34
|
+
## Next milestones
|
|
35
|
+
|
|
36
|
+
1. **0.4**: JSON Schema files validated in CI, SARIF export, `import-traces` for OpenTelemetry JSON.
|
|
37
|
+
2. **0.5**: property-based tests for parsers and scorer, signed capsule option.
|
|
38
|
+
3. **0.6**: sandboxed local runner behind explicit consent, budgets and timeouts.
|
|
39
|
+
4. **0.7**: MCP Inspector and Harbor adapters as optional packages.
|
|
40
|
+
5. **1.0**: only when live-host compatibility is tested and the report schema is stable.
|
|
41
|
+
|
|
42
|
+
## Decisions deferred
|
|
43
|
+
|
|
44
|
+
Which single observability UI to support first (Opik or Phoenix; Phoenix licence differs). Whether to adopt Changesets or Release Please. DuckDB or SQLite storage (only if flat files stop being enough).
|
|
45
|
+
|
|
46
|
+
## How to propose a change
|
|
47
|
+
|
|
48
|
+
Open an issue with the problem, the evidence you can supply, and the test that would prove it works.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 470" role="img" aria-labelledby="t d">
|
|
2
|
+
<title id="t">EvalSutra architecture</title>
|
|
3
|
+
<desc id="d">Inputs (suite and observations JSON) flow through validation, deterministic graders, scoring and gate, then reporters and the repro capsule. CLI sits on top, skills sit beside it. No network.</desc>
|
|
4
|
+
<style>
|
|
5
|
+
.bx{fill:#fff;stroke:#4b3fd6;stroke-width:1.6}.bg{fill:#eef0f7;stroke:#8b93a8;stroke-width:1.2}.tx{font:600 14px system-ui,sans-serif;fill:#14161f}.sm{font:12px system-ui,sans-serif;fill:#566074}.ar{stroke:#4b3fd6;stroke-width:2;fill:none;marker-end:url(#a)}.no{fill:#fde8e8;stroke:#a4161a;stroke-width:1.4}
|
|
6
|
+
@media (prefers-color-scheme:dark){.bx{fill:#161a24;stroke:#8f86ff}.bg{fill:#1d2232;stroke:#4a5370}.tx{fill:#eceff7}.sm{fill:#9aa4ba}.ar{stroke:#8f86ff}.no{fill:#3a1618;stroke:#ff8d90}}
|
|
7
|
+
</style>
|
|
8
|
+
<defs><marker id="a" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10z" fill="#4b3fd6"/></marker></defs>
|
|
9
|
+
<rect class="bg" x="20" y="20" width="860" height="60" rx="12"/><text class="tx" x="40" y="48">CLI packages/cli/evalsutra.mjs</text><text class="sm" x="40" y="68">doctor · init · validate · run · compare · report · gate · audit · trace · repro · install · uninstall · list-skills · matrix</text>
|
|
10
|
+
<rect class="bx" x="20" y="120" width="150" height="70" rx="10"/><text class="tx" x="36" y="150">Suite JSON</text><text class="sm" x="36" y="170">cases + assertions</text>
|
|
11
|
+
<rect class="bx" x="20" y="220" width="150" height="70" rx="10"/><text class="tx" x="36" y="250">Observations</text><text class="sm" x="36" y="270">output, tools, latency</text>
|
|
12
|
+
<rect class="bx" x="230" y="150" width="170" height="90" rx="10"/><text class="tx" x="246" y="180">Validate</text><text class="sm" x="246" y="200">schema, ids, regex guard</text><text class="sm" x="246" y="218">typed errors, exit 4</text>
|
|
13
|
+
<rect class="bx" x="460" y="120" width="190" height="150" rx="10"/><text class="tx" x="476" y="150">Deterministic core</text><text class="sm" x="476" y="172">graders (pure functions)</text><text class="sm" x="476" y="190">evidence hierarchy</text><text class="sm" x="476" y="208">weighted score + coverage</text><text class="sm" x="476" y="226">Wilson CI · sign test</text><text class="sm" x="476" y="244">gate NO_GO/REVIEW/GO_CANDIDATE</text>
|
|
14
|
+
<rect class="bx" x="710" y="120" width="170" height="70" rx="10"/><text class="tx" x="726" y="150">Reporters</text><text class="sm" x="726" y="170">JSON · MD · HTML · JUnit</text>
|
|
15
|
+
<rect class="bx" x="710" y="220" width="170" height="70" rx="10"/><text class="tx" x="726" y="250">Repro capsule</text><text class="sm" x="726" y="270">SHA-256 manifest, redacted</text>
|
|
16
|
+
<rect class="bx" x="230" y="320" width="170" height="70" rx="10"/><text class="tx" x="246" y="350">Trace analysis</text><text class="sm" x="246" y="370">loops, forbidden, claims</text>
|
|
17
|
+
<rect class="bx" x="460" y="320" width="190" height="70" rx="10"/><text class="tx" x="476" y="350">Self-audit</text><text class="sm" x="476" y="370">29 checks → MD + HTML</text>
|
|
18
|
+
<rect class="bg" x="710" y="320" width="170" height="70" rx="10"/><text class="tx" x="726" y="350">18 agent skills</text><text class="sm" x="726" y="370">run by the host AI</text>
|
|
19
|
+
<rect class="no" x="20" y="420" width="860" height="36" rx="10"/><text class="tx" x="40" y="443">Out of scope in v0.2: network calls · live agent execution · model inference · server · telemetry</text>
|
|
20
|
+
<path class="ar" d="M170 155 L230 185"/><path class="ar" d="M170 255 L230 215"/><path class="ar" d="M400 195 L460 195"/><path class="ar" d="M650 155 L710 155"/><path class="ar" d="M650 245 L710 255"/><path class="ar" d="M400 350 L460 350"/><path class="ar" d="M120 80 L120 120"/>
|
|
21
|
+
</svg>
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
# Evidence
|
|
2
|
+
|
|
3
|
+
Generated by `node scripts/build-evidence.mjs` on Node v24.15.0 (win32 x64). Nothing here is hand-written.
|
|
4
|
+
|
|
5
|
+
## Test run
|
|
6
|
+
|
|
7
|
+
| Tests | Passed | Failed | Skipped | Cancelled | Exit code |
|
|
8
|
+
|---|---|---|---|---|---|
|
|
9
|
+
| 57 | 56 | 0 | 1 | 0 | 0 |
|
|
10
|
+
|
|
11
|
+
Full output: [test-run.txt](test-run.txt). Machine-readable: [test-summary.json](test-summary.json).
|
|
12
|
+
|
|
13
|
+
Skipped tests are POSIX-only symlink checks that cannot run on Windows. Linux and macOS results come from the public GitHub Actions runs, not from this file.
|
|
14
|
+
|
|
15
|
+
## Sample runs: exit codes behave as documented
|
|
16
|
+
|
|
17
|
+
| Command | Exit code | Documented | Matches |
|
|
18
|
+
|---|---|---|---|
|
|
19
|
+
| `run (good evidence)` | 0 | 0 | yes |
|
|
20
|
+
| `run (bad evidence)` | 2 | 2 | yes |
|
|
21
|
+
| `compare good vs bad` | 2 | 2 | yes |
|
|
22
|
+
| `gate bad report` | 2 | 2 | yes |
|
|
23
|
+
| `trace analyze bad trace` | 2 | 2 | yes |
|
|
24
|
+
| `audit` | 0 | 0 | yes |
|
|
25
|
+
|
|
26
|
+
## Sample artifacts
|
|
27
|
+
|
|
28
|
+
- Good evidence: [report.json](samples/good/report.json), [report.md](samples/good/report.md), [report.html](samples/good/report.html), [junit.xml](samples/good/junit.xml)
|
|
29
|
+
- Bad evidence (forbidden tool + canary leak, missing latency): [report.json](samples/bad/report.json), [report.md](samples/bad/report.md), [report.html](samples/bad/report.html), [junit.xml](samples/bad/junit.xml)
|
|
30
|
+
- Regression comparison: [compare-good-vs-bad.json](samples/compare-good-vs-bad.json)
|
|
31
|
+
- Release gate on the bad run: [gate-bad-run.json](samples/gate-bad-run.json)
|
|
32
|
+
- Trace analysis: [trace-analysis.json](samples/trace-analysis.json)
|
|
33
|
+
- Self-audit: [audit-report.md](samples/audit/audit-report.md), [audit-report.html](samples/audit/audit-report.html), [audit-report.json](samples/audit/audit-report.json)
|
|
34
|
+
|
|
35
|
+
## Honest limits
|
|
36
|
+
|
|
37
|
+
- Observations are **synthetic fixtures**. They show the evaluator behaves correctly; they say nothing about any real agent.
|
|
38
|
+
- The self-audit runs in the same process as the code it checks. It is an integrity check, not a third-party attestation.
|
|
39
|
+
- No live IDE, model or cloud service was used to produce any file here.
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
<!doctype html>
|
|
2
|
+
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
3
|
+
<title>EvalSutra self-audit report</title><meta name="description" content="Machine-generated self-audit of EvalSutra: grader accuracy, reproducibility, security and legal checks."><meta name="author" content="Vatsal Shah"><meta name="color-scheme" content="light dark">
|
|
4
|
+
<meta property="og:title" content="EvalSutra self-audit report"><meta property="og:description" content="Machine-generated self-audit of EvalSutra: grader accuracy, reproducibility, security and legal checks."><meta property="og:type" content="website">
|
|
5
|
+
<style>
|
|
6
|
+
:root{--bg:#f7f8fb;--surface:#fff;--ink:#14161f;--muted:#566074;--line:#d9deea;--brand:#4b3fd6;--brand-ink:#fff;--ok:#0b6b3a;--ok-bg:#e3f6ea;--bad:#a4161a;--bad-bg:#fde8e8;--warn:#7a4b00;--warn-bg:#fff2d6;--info:#124e8a;--info-bg:#e4f0fc;--code:#eef0f7;--radius:14px;--shadow:0 1px 2px rgba(20,22,31,.06),0 8px 24px rgba(20,22,31,.06)}
|
|
7
|
+
@media (prefers-color-scheme:dark){:root{--bg:#0e1017;--surface:#161a24;--ink:#eceff7;--muted:#9aa4ba;--line:#2a3042;--brand:#8f86ff;--brand-ink:#0e1017;--ok:#6fe0a0;--ok-bg:#12301f;--bad:#ff8d90;--bad-bg:#3a1618;--warn:#ffd27a;--warn-bg:#3a2a0a;--info:#8cc4ff;--info-bg:#10294a;--code:#1d2232;--shadow:none}}
|
|
8
|
+
*{box-sizing:border-box}html{scroll-behavior:smooth}
|
|
9
|
+
body{margin:0;background:var(--bg);color:var(--ink);font:16px/1.65 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;-webkit-font-smoothing:antialiased}
|
|
10
|
+
a{color:var(--brand)}a:hover{text-decoration-thickness:2px}
|
|
11
|
+
:focus-visible{outline:3px solid var(--brand);outline-offset:2px;border-radius:4px}
|
|
12
|
+
.skip{position:absolute;left:-999px;top:0;background:var(--brand);color:var(--brand-ink);padding:.6rem 1rem;z-index:9}.skip:focus{left:0}
|
|
13
|
+
header.site{border-bottom:1px solid var(--line);background:var(--surface);position:sticky;top:0;z-index:5}
|
|
14
|
+
header.site div{max-width:72rem;margin:auto;padding:.7rem 1.25rem;display:flex;gap:1rem;align-items:center;flex-wrap:wrap}
|
|
15
|
+
header.site strong{font-size:1.05rem;letter-spacing:.2px}header.site nav{display:flex;gap:.25rem 1rem;flex-wrap:wrap;margin-left:auto}header.site nav a{text-decoration:none;color:var(--muted);font-size:.92rem;padding:.2rem .1rem}header.site nav a[aria-current=page],header.site nav a:hover{color:var(--ink);box-shadow:0 2px 0 var(--brand)}
|
|
16
|
+
main{max-width:72rem;margin:0 auto;padding:2rem 1.25rem 4rem}
|
|
17
|
+
.hero{background:linear-gradient(135deg,var(--brand),#7b5cff 60%,#00a3a3);color:#fff;border-radius:calc(var(--radius) + 6px);padding:2.4rem 2rem;box-shadow:var(--shadow);margin-bottom:2rem}
|
|
18
|
+
.hero h1{margin:.1rem 0 .4rem;font-size:clamp(1.7rem,4vw,2.6rem);line-height:1.15}.hero p{margin:.2rem 0;max-width:46rem;opacity:.95}
|
|
19
|
+
.hero .tag{font-weight:600;letter-spacing:.3px}
|
|
20
|
+
h1,h2,h3{line-height:1.25}h2{margin-top:2.4rem;padding-bottom:.35rem;border-bottom:1px solid var(--line)}h3{margin-top:1.6rem}
|
|
21
|
+
.grid{display:grid;gap:1rem;grid-template-columns:repeat(auto-fit,minmax(15rem,1fr));margin:1rem 0}
|
|
22
|
+
.card{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:1.1rem 1.2rem;box-shadow:var(--shadow)}
|
|
23
|
+
.card h3{margin:.1rem 0 .4rem;font-size:1.05rem}.card p{margin:.3rem 0;color:var(--muted)}
|
|
24
|
+
.stat{font-size:2rem;font-weight:700;line-height:1.1}.stat small{display:block;font-size:.8rem;font-weight:500;color:var(--muted);margin-top:.3rem}
|
|
25
|
+
table{border-collapse:collapse;width:100%;background:var(--surface);border:1px solid var(--line);border-radius:10px;overflow:hidden;font-size:.95rem}
|
|
26
|
+
caption{text-align:left;padding:.5rem 0;color:var(--muted);font-size:.9rem}
|
|
27
|
+
th,td{padding:.55rem .75rem;border-bottom:1px solid var(--line);text-align:left;vertical-align:top}thead th{background:var(--code);font-size:.85rem;text-transform:uppercase;letter-spacing:.4px}tbody tr:last-child td,tbody tr:last-child th{border-bottom:0}
|
|
28
|
+
.tablewrap{overflow-x:auto;margin:1rem 0}
|
|
29
|
+
code,pre{font:.9em/1.5 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}code{background:var(--code);padding:.1rem .35rem;border-radius:5px}
|
|
30
|
+
pre{background:var(--code);padding:1rem;border-radius:10px;overflow-x:auto;border:1px solid var(--line)}pre code{background:none;padding:0}
|
|
31
|
+
blockquote{margin:1rem 0;padding:.6rem 1rem;border-left:4px solid var(--brand);background:var(--info-bg);border-radius:0 10px 10px 0}
|
|
32
|
+
.badge{display:inline-flex;align-items:center;gap:.3rem;padding:.12rem .6rem;border-radius:999px;font-size:.82rem;font-weight:600;border:1px solid currentColor;white-space:nowrap}
|
|
33
|
+
.b-pass,.b-go_candidate{color:var(--ok);background:var(--ok-bg)}.b-fail,.b-no_go{color:var(--bad);background:var(--bad-bg)}.b-inconclusive,.b-review,.b-error,.b-not_supported,.b-skipped{color:var(--warn);background:var(--warn-bg)}.b-info{color:var(--info);background:var(--info-bg)}
|
|
34
|
+
footer{max-width:72rem;margin:0 auto;padding:1.5rem 1.25rem 3rem;color:var(--muted);font-size:.88rem;border-top:1px solid var(--line)}
|
|
35
|
+
.toc{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:1rem 1.3rem;margin:1rem 0}.toc ul{margin:.3rem 0;padding-left:1.2rem}
|
|
36
|
+
@media (max-width:40rem){header.site nav{margin-left:0}.hero{padding:1.5rem 1.1rem}}
|
|
37
|
+
@media print{header.site,.skip{display:none}body{background:#fff;color:#000}.card,table{box-shadow:none}a{color:#000}}
|
|
38
|
+
@media (prefers-reduced-motion:reduce){html{scroll-behavior:auto}}
|
|
39
|
+
</style></head><body><a class="skip" href="#main">Skip to content</a>
|
|
40
|
+
<header class="site"><div><strong>EvalSutra AI</strong></div></header>
|
|
41
|
+
<main id="main"><section class="hero"><p class="tag">Independent self-audit</p><h1>EvalSutra audit: PASS</h1><p>29 of 29 checks passed. Self-audit of the EvalSutra package files. It audits the tool, not any third-party agent.</p><p><small>evalsutra@0.4.0 · Node v24.15.0 · win32</small></p></section>
|
|
42
|
+
<h2>Summary by category</h2><div class="grid"><div class="card"><h3>Grader accuracy</h3><div class="stat">3/3<small>checks passed</small></div></div><div class="card"><h3>Evidence rules</h3><div class="stat">7/7<small>checks passed</small></div></div><div class="card"><h3>Reproducibility</h3><div class="stat">3/3<small>checks passed</small></div></div><div class="card"><h3>Trace analysis</h3><div class="stat">2/2<small>checks passed</small></div></div><div class="card"><h3>Security</h3><div class="stat">6/6<small>checks passed</small></div></div><div class="card"><h3>Skills</h3><div class="stat">4/4<small>checks passed</small></div></div><div class="card"><h3>Legal</h3><div class="stat">4/4<small>checks passed</small></div></div></div>
|
|
43
|
+
<h2>All checks</h2><div class="tablewrap"><table><caption>Every check with expected and observed value. Result is text plus symbol.</caption><thead><tr><th>ID</th><th>Category</th><th>Check</th><th>Expected</th><th>Observed</th><th>Result</th></tr></thead><tbody><tr><th scope="row">G1</th><td>Grader accuracy</td><td>Known-answer table (36 cases)<br><small>all rows matched ground truth</small></td><td>0 wrong</td><td>0 wrong</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">G2</th><td>Grader accuracy</td><td>False positives (graded pass, truth not pass)</td><td>0</td><td>0</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">G3</th><td>Grader accuracy</td><td>False negatives (graded not-pass, truth pass)</td><td>0</td><td>0</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">E1</th><td>Evidence rules</td><td>No observations at all must never pass</td><td>REVIEW, passRate null, coverage 0</td><td>REVIEW, null, 0</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">E2</th><td>Evidence rules</td><td>Model-judge-only evidence cannot reach GO_CANDIDATE</td><td>not GO_CANDIDATE</td><td>REVIEW</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">E3</th><td>Evidence rules</td><td>All-skipped suite must not pass</td><td>not GO_CANDIDATE, 0 passed</td><td>REVIEW, 0 passed</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">E4</th><td>Evidence rules</td><td>Forbidden tool + canary leak is NO_GO with a critical failure</td><td>NO_GO, 1 critical</td><td>NO_GO, 1 critical</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">E5</th><td>Evidence rules</td><td>Clean demo evidence is GO_CANDIDATE (and only a candidate)</td><td>GO_CANDIDATE</td><td>GO_CANDIDATE</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">E6</th><td>Evidence rules</td><td>Regression between good and bad run is detected</td><td>verdict FAIL</td><td>FAIL</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">E7</th><td>Evidence rules</td><td>Case id "constructor" does not read inherited object members</td><td>inconclusive</td><td>inconclusive</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">R1</th><td>Reproducibility</td><td>Same inputs give byte-identical results</td><td>equal</td><td>equal</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">R2</th><td>Reproducibility</td><td>Input digest ignores JSON key order</td><td>equal</td><td>equal</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">R3</th><td>Reproducibility</td><td>Changing one value changes the digest</td><td>different</td><td>different</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">T1</th><td>Trace analysis</td><td>Success claim after failed last tool is flagged</td><td>UNSUPPORTED_SUCCESS_CLAIM</td><td>UNSUPPORTED_SUCCESS_CLAIM</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">T2</th><td>Trace analysis</td><td>Empty trace is inconclusive, never pass</td><td>inconclusive</td><td>inconclusive</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">S1</th><td>Security</td><td>Report HTML/XML escape hostile suite names</td><td>no raw <img</td><td>escaped</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">S2</th><td>Security</td><td>Shell metacharacters in fixtures are inert data</td><td>valid, nothing executed</td><td>valid</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">S3</th><td>Security</td><td>Secret scan over 138 text files</td><td>0 files with credential patterns</td><td>0</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">S4</th><td>Security</td><td>No network/process/dynamic-code APIs in 10 source files</td><td>0 hits</td><td>0</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">S5</th><td>Security</td><td>Zero runtime dependencies (no supply-chain surface)</td><td>0</td><td>0</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">S6</th><td>Security</td><td>No install-time lifecycle scripts</td><td>0</td><td>0</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">K1</th><td>Skills</td><td>Eighteen specialist skills present</td><td>18</td><td>18</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">K2</th><td>Skills</td><td>Valid frontmatter (name matches folder, description present)</td><td>0 invalid</td><td>0</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">K3</th><td>Skills</td><td>Every skill states objective, procedure, contract and stop conditions</td><td>0 missing</td><td>0</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">K4</th><td>Skills</td><td>Plugin copy equals canonical copy</td><td>0 drift</td><td>0</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">L1</th><td>Legal</td><td>Apache-2.0 LICENSE text present</td><td>contains "Apache License" and "Version 2.0"</td><td>yes</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">L2</th><td>Legal</td><td>package.json license is Apache-2.0</td><td>Apache-2.0</td><td>Apache-2.0</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">L3</th><td>Legal</td><td>NOTICE and THIRD_PARTY_NOTICES present</td><td>both exist</td><td>true/true</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr><tr><th scope="row">L4</th><td>Legal</td><td>SECURITY, CONTRIBUTING, CODE_OF_CONDUCT, CHANGELOG present</td><td>all exist</td><td>checked</td><td><span class="badge b-pass"><span aria-hidden="true">✔</span> PASS</span></td></tr></tbody></table></div>
|
|
44
|
+
<h2>Limits of this audit</h2><ul><li>Runs in-process with the code it audits, so it is a regression and integrity check, not an independent third-party attestation.</li><li>Known-answer table is finite; it proves listed behaviours only.</li><li>Does not test live IDEs, live models or other operating systems; see CI for OS results.</li></ul>
|
|
45
|
+
<h2>Reproduce</h2><pre><code>npx evalsutra audit --out audit</code></pre></main>
|
|
46
|
+
<footer>EvalSutra AI · Created by Vatsal Shah · Apache-2.0 · Generated offline; no scripts, no trackers, no network calls.</footer></body></html>
|