evalsutra 0.0.0-stage → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/evalsutra-arena/SKILL.md +27 -0
- package/.agents/skills/evalsutra-audit/SKILL.md +27 -0
- package/.agents/skills/evalsutra-bench/SKILL.md +27 -0
- package/.agents/skills/evalsutra-cost/SKILL.md +27 -0
- package/.agents/skills/evalsutra-data/SKILL.md +27 -0
- package/.agents/skills/evalsutra-drift/SKILL.md +27 -0
- package/.agents/skills/evalsutra-explain/SKILL.md +27 -0
- package/.agents/skills/evalsutra-guard/SKILL.md +27 -0
- package/.agents/skills/evalsutra-judge/SKILL.md +27 -0
- package/.agents/skills/evalsutra-pluginwatch/SKILL.md +27 -0
- package/.agents/skills/evalsutra-port/SKILL.md +27 -0
- package/.agents/skills/evalsutra-prime/SKILL.md +27 -0
- package/.agents/skills/evalsutra-report/SKILL.md +27 -0
- package/.agents/skills/evalsutra-robust/SKILL.md +27 -0
- package/.agents/skills/evalsutra-task/SKILL.md +27 -0
- package/.agents/skills/evalsutra-trace/SKILL.md +27 -0
- package/.agents/skills/evalsutra-truth/SKILL.md +27 -0
- package/.agents/skills/evalsutra-verify/SKILL.md +27 -0
- package/.claude-plugin/plugin.json +8 -0
- package/CHANGELOG.md +33 -0
- package/CITATION.cff +14 -0
- package/CODE_OF_CONDUCT.md +2 -0
- package/CONTRIBUTING.md +2 -0
- package/LICENSE +202 -0
- package/NOTICE +5 -0
- package/README.md +161 -2
- package/SECURITY.md +17 -0
- package/SUPPORT.md +5 -0
- package/THIRD_PARTY_NOTICES.md +14 -0
- package/docs/ADVANCED_TOOLING_ROADMAP.md +244 -0
- package/docs/ARCHITECTURE.md +17 -0
- package/docs/PLUGIN_AND_SKILL_CATALOG.md +56 -0
- package/docs/STATUS.md +25 -0
- package/docs/master-claude-prompt.md +534 -0
- package/documents/evalsutra/01-executive-summary.md +52 -0
- package/documents/evalsutra/02-functional-specification.md +104 -0
- package/documents/evalsutra/03-architecture.md +78 -0
- package/documents/evalsutra/04-technical-reference.md +150 -0
- package/documents/evalsutra/05-user-guide.md +94 -0
- package/documents/evalsutra/06-skills-catalog.md +52 -0
- package/documents/evalsutra/07-security-and-legal.md +66 -0
- package/documents/evalsutra/08-testing-and-evidence.md +63 -0
- package/documents/evalsutra/09-audit-and-trust.md +54 -0
- package/documents/evalsutra/10-ci-cd-and-release.md +50 -0
- package/documents/evalsutra/11-faq-and-glossary.md +51 -0
- package/documents/evalsutra/12-contributor-handbook.md +67 -0
- package/documents/evalsutra/13-roadmap-and-status.md +48 -0
- package/documents/evalsutra/assets/architecture.svg +21 -0
- package/documents/evalsutra/evidence/README.md +39 -0
- package/documents/evalsutra/evidence/samples/audit/audit-report.html +46 -0
- package/documents/evalsutra/evidence/samples/audit/audit-report.json +317 -0
- package/documents/evalsutra/evidence/samples/audit/audit-report.md +64 -0
- package/documents/evalsutra/evidence/samples/bad/junit.xml +2 -0
- package/documents/evalsutra/evidence/samples/bad/report.html +47 -0
- package/documents/evalsutra/evidence/samples/bad/report.json +153 -0
- package/documents/evalsutra/evidence/samples/bad/report.md +22 -0
- package/documents/evalsutra/evidence/samples/compare-good-vs-bad.json +59 -0
- package/documents/evalsutra/evidence/samples/demo-suite.json +13 -0
- package/documents/evalsutra/evidence/samples/gate-bad-run.json +10 -0
- package/documents/evalsutra/evidence/samples/good/junit.xml +2 -0
- package/documents/evalsutra/evidence/samples/good/report.html +47 -0
- package/documents/evalsutra/evidence/samples/good/report.json +153 -0
- package/documents/evalsutra/evidence/samples/good/report.md +22 -0
- package/documents/evalsutra/evidence/samples/trace-analysis.json +33 -0
- package/documents/evalsutra/evidence/test-run.txt +71 -0
- package/documents/evalsutra/evidence/test-summary.json +11 -0
- package/documents/evalsutra/html/01-executive-summary.html +57 -0
- package/documents/evalsutra/html/02-functional-specification.html +75 -0
- package/documents/evalsutra/html/03-architecture.html +60 -0
- package/documents/evalsutra/html/04-technical-reference.html +109 -0
- package/documents/evalsutra/html/05-user-guide.html +72 -0
- package/documents/evalsutra/html/06-skills-catalog.html +54 -0
- package/documents/evalsutra/html/07-security-and-legal.html +63 -0
- package/documents/evalsutra/html/08-testing-and-evidence.html +61 -0
- package/documents/evalsutra/html/09-audit-and-trust.html +61 -0
- package/documents/evalsutra/html/10-ci-cd-and-release.html +61 -0
- package/documents/evalsutra/html/11-faq-and-glossary.html +57 -0
- package/documents/evalsutra/html/12-contributor-handbook.html +71 -0
- package/documents/evalsutra/html/13-roadmap-and-status.html +51 -0
- package/documents/evalsutra/html/assets/architecture.svg +21 -0
- package/documents/evalsutra/html/index.html +42 -0
- package/examples/demo-observations-bad.json +7 -0
- package/examples/demo-observations.json +7 -0
- package/examples/observations.json +10 -0
- package/examples/output/report.json +99 -0
- package/examples/output/report.md +18 -0
- package/examples/suite.json +11 -0
- package/examples/traces/bad-trace.json +5 -0
- package/llms.txt +25 -0
- package/package.json +19 -4
- package/packages/audit/audit.mjs +179 -0
- package/packages/cli/evalsutra.mjs +151 -0
- package/packages/cli/install.mjs +68 -0
- package/packages/core/capsule.mjs +31 -0
- package/packages/core/evaluate.mjs +187 -0
- package/packages/core/security.mjs +22 -0
- package/packages/core/stats.mjs +16 -0
- package/packages/core/trace.mjs +29 -0
- package/packages/reporters/index.mjs +26 -0
- package/packages/reporters/theme.mjs +53 -0
- package/policies/README.md +3 -0
- package/policies/default.json +1 -0
- package/skills/evalsutra-arena/SKILL.md +27 -0
- package/skills/evalsutra-audit/SKILL.md +27 -0
- package/skills/evalsutra-bench/SKILL.md +27 -0
- package/skills/evalsutra-cost/SKILL.md +27 -0
- package/skills/evalsutra-data/SKILL.md +27 -0
- package/skills/evalsutra-drift/SKILL.md +27 -0
- package/skills/evalsutra-explain/SKILL.md +27 -0
- package/skills/evalsutra-guard/SKILL.md +27 -0
- package/skills/evalsutra-judge/SKILL.md +27 -0
- package/skills/evalsutra-pluginwatch/SKILL.md +27 -0
- package/skills/evalsutra-port/SKILL.md +27 -0
- package/skills/evalsutra-prime/SKILL.md +27 -0
- package/skills/evalsutra-report/SKILL.md +27 -0
- package/skills/evalsutra-robust/SKILL.md +27 -0
- package/skills/evalsutra-task/SKILL.md +27 -0
- package/skills/evalsutra-trace/SKILL.md +27 -0
- package/skills/evalsutra-truth/SKILL.md +27 -0
- package/skills/evalsutra-verify/SKILL.md +27 -0
- package/suites/demo.json +13 -0
|
@@ -0,0 +1,534 @@
|
|
|
1
|
+
# EVALSUTRA AI — MASTER ENGINEERING PROMPT FOR CLAUDE
|
|
2
|
+
|
|
3
|
+
**Working product:** EvalSutra AI — The Agent Evaluation & Reliability Intelligence Platform
|
|
4
|
+
**Tagline:** Test Intelligence. Measure Truth. Build Trust.
|
|
5
|
+
**Creator/publisher attribution:** Vatsal Shah
|
|
6
|
+
**Proposed GitHub repository and CLI:** `evalsutra`
|
|
7
|
+
**Status:** Open-source product specification; implement and verify. Not a claim of completed functionality.
|
|
8
|
+
**License proposal:** Apache License 2.0 (verify dependency compatibility).
|
|
9
|
+
**Primary goal:** A trustworthy, open, extensible, evidence-first evaluation platform for AI agents and agentic workflows that works offline with recorded traces, inside supported AI IDEs using the IDE's own AI, and optionally with independently authenticated local/cloud/server runtimes.
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
## 0. Your mission and working rules
|
|
14
|
+
|
|
15
|
+
You are acting as a coordinated team of principal AI evaluation engineers, software architects, SREs, security architects, developer-experience leads, test engineers, AI safety specialists, applied statisticians, technical writers, and open-source maintainers.
|
|
16
|
+
|
|
17
|
+
**Build the actual project in the present workspace, inside the one `evalsutra/` root folder.** You must inspect any existing starter files, preserve tested functionality, run the relevant tests, implement working source code, and produce accurate progress and acceptance evidence. Do not deliver only suggestions, a design, or an empty scaffold. Work through coherent vertical slices that run from CLI to core engine to tests to reports. Do not claim you built features that are only documented.
|
|
18
|
+
|
|
19
|
+
If the starter exists, read `README.md`, `AGENTS.md`, `CLAUDE.md`, `docs/` and `packages/`, run `npm test`, and select the next most valuable missing feature. Maintain backward compatibility or explain migrations. Ask only when essential authorization or irreversible choices are missing. Never auto-push, publish, deploy, install globally, open network ports, access secrets, or purchase services.
|
|
20
|
+
|
|
21
|
+
Before each major implementation: inspect, plan the smallest secure slice, implement, test, and update documentation and the feature matrix. **Quality evidence is more important than feature-count claims.**
|
|
22
|
+
|
|
23
|
+
## 1. Product identity, distinctiveness and value
|
|
24
|
+
|
|
25
|
+
Build **EvalSutra AI** as an independent open-source tool that can also audit the effectiveness of related projects ArchSutra AI (architecture), VigilSutra AI (security), RachnaSutra AI (UX), and PramanaSutra AI (QA). No hard dependency on any sibling project. Attribute creator as `Vatsal Shah`; preserve contributor credit and use non-misleading authorship. Use an internationally accessible, simple CLI; accessibility-aware optional dashboard; high-quality docs, diagrams, demos, badges from real CI, versioned schemas and release notes.
|
|
26
|
+
|
|
27
|
+
Differentiators:
|
|
28
|
+
1. Outcomes-first scoring: evaluate real end state and independent test evidence; never grade prose as completed work.
|
|
29
|
+
2. Agent portability lab: measure feature support and behavior across IDEs/operating systems/providers using verified adapters.
|
|
30
|
+
3. Trace truthfulness: distinguish executed tool traces, imported logs, human annotations, AI judgments and missing data.
|
|
31
|
+
4. Reproducibility: immutable fixture hashes, seeds, budgets, versions, evaluation config, environment and timestamps.
|
|
32
|
+
5. Plugin/skill auditing: package integrity, manifest compatibility, unexpected permissions and prompt-injection resilience.
|
|
33
|
+
6. Local-first zero-key mode, with hosted/provider integrations opt-in.
|
|
34
|
+
7. Red-team-ready safe synthetic fixtures, no attacks against uncontrolled real targets.
|
|
35
|
+
8. Regression intelligence: compare baselines under controlled conditions and explain uncertainty and confounders.
|
|
36
|
+
|
|
37
|
+
## 2. Mandatory operating modes
|
|
38
|
+
|
|
39
|
+
**MODE A — Offline deterministic replay (mandatory and always available):** Consume local user-provided fixtures, outputs, traces, metadata and artifacts. Run static/deterministic graders without API keys or model access; produce accurate provenance and coverage. No simulated LLM execution disguised as real agent results.
|
|
40
|
+
|
|
41
|
+
**MODE B — IDE-native (mandatory adapters, feature-limited per host):** Reusable `SKILL.md`, project instructions, optionally subagents/commands/hooks appropriate to each host. Supported IDE inference remains inside that IDE's authenticated capabilities. A PowerShell/bash launcher does not magically gain access to the IDE's inference runtime. Use documented host SDK/CLI/workflow when live execution is available, otherwise guide the user through manual evaluation or trace import. Do not circumvent subscriptions or authentication.
|
|
42
|
+
|
|
43
|
+
**MODE C — Local model (optional):** Ollama or equivalent through explicitly enabled adapter; local inference hardware and model downloads are the user's responsibility. Do not claim benchmark quality is the same as hosted models.
|
|
44
|
+
|
|
45
|
+
**MODE D — Server/API (optional):** Authorized OpenAI, Anthropic/Claude, Google/Gemini, other supported APIs, or supported self-hosted inference. Use credentials through secure environment/secret manager at runtime; never commit secrets or send user data without consent. Respect provider license, ToS, rate limits, quotas, concurrency caps and cost budgets. Cloud billing must not be enabled implicitly.
|
|
46
|
+
|
|
47
|
+
**MODE E — CI/CD or air-gapped worker (optional):** Deterministic data-only evaluation and quality gates run without cloud AI. For live evaluation, require explicit runner enablement and isolation, pinned dependencies and budgets.
|
|
48
|
+
|
|
49
|
+
## 3. Host compatibility contract — tested not advertised
|
|
50
|
+
|
|
51
|
+
Targets: Claude Code, Cursor, OpenAI Codex, Gemini CLI, GitHub Copilot (VS Code and GitHub environments where available), generic Agent Skills-capable clients, headless CLI, Linux server and Windows/macOS/Linux terminals. Explore additional environments via plugin contributions rather than a claim of universal support.
|
|
52
|
+
|
|
53
|
+
Canonical skills: `.agents/skills/<skill-name>/SKILL.md`, with YAML `name` and `description` frontmatter. For Claude Code project-local `.claude/skills/`; Cursor `.cursor/skills/` or `.agents/skills/`; Codex `.agents/skills/` with host-version verification; Gemini CLI `.gemini/skills/` or `.agents/skills/`; Copilot `.github/skills/` or `.agents/skills/`. Implement adapters separately, verify host-specific changes against official current docs, and maintain a compatibility matrix: skill discovery, tool access, trace export, headless execution, subagents, runtime permissions, model metadata, test status and limitations.
|
|
54
|
+
|
|
55
|
+
Prefer a central source of agent skill content with deterministic generation/copying to targets. Implement `evalsutra install --ide ... --project ...` as an idempotent, consentful, project-local operation: preview changes, never clobber existing files, support uninstallation with change detection, prevent path traversal and symlink escape, and avoid global side effects. For a platform that cannot execute agents headlessly, offer manual evaluation, not fake automation.
|
|
56
|
+
|
|
57
|
+
## 4. Agent roster and role contracts
|
|
58
|
+
|
|
59
|
+
Build modular specialist skills and (when documented) real host subagent configurations, not mandatory separate model calls:
|
|
60
|
+
- **EvalSutra Prime:** evaluation plan, scope authorization, orchestration, consolidated release assessment.
|
|
61
|
+
- **EvalSutra Bench:** benchmark suite authoring, controlled fixture generation, split and version management.
|
|
62
|
+
- **EvalSutra Truth:** check answers against trusted reference material and objective ground truth; label subjective cases.
|
|
63
|
+
- **EvalSutra Trace:** tool action sequence, tool argument envelopes, timing and noncompliant actions.
|
|
64
|
+
- **EvalSutra Task:** independently inspect task end state, filesystem/database/browser artifacts and acceptance tests.
|
|
65
|
+
- **EvalSutra Guard:** synthetic prompt-injection, privacy and permissions boundary tests in isolated environments.
|
|
66
|
+
- **EvalSutra Robust:** failures, interruptions, timeouts, retries, partial completion and recovery.
|
|
67
|
+
- **EvalSutra Drift:** temporal regression and variant behavior changes under controlled conditions.
|
|
68
|
+
- **EvalSutra Port:** IDE/provider/OS compatibility audits and platform feature detection.
|
|
69
|
+
- **EvalSutra Judge:** optional calibrated LLM-as-judge with human adjudication, blind review and bias detection.
|
|
70
|
+
- **EvalSutra Cost:** token usage, latency distribution, throughput, budget tracking with explicit units.
|
|
71
|
+
- **EvalSutra Data:** dataset governance, leakage/contamination detection, provenance and licensing.
|
|
72
|
+
- **EvalSutra Arena:** fair paired comparisons; never imply universal rankings from a limited dataset.
|
|
73
|
+
- **EvalSutra Explain:** root cause, evidence pointers, confidence and actionable remediation.
|
|
74
|
+
- **EvalSutra Audit:** grader validation, reproducibility and evaluation integrity checks.
|
|
75
|
+
- **EvalSutra Verify:** independently re-run tests and release quality gates.
|
|
76
|
+
- **EvalSutra Report:** JSON, Markdown, HTML, SARIF where appropriate, machine-consumable and human-readable reports.
|
|
77
|
+
- **EvalSutra PluginWatch:** evaluate third-party skill/plugin package provenance and requested capabilities without automatically executing plugin code.
|
|
78
|
+
|
|
79
|
+
Each agent must have `SKILL.md` with use cases, inputs, outputs, allowed tools, explicit permission checks, failure handling, stop conditions, and handoff artifact contract. Build tests for at least representative examples. Maintain honest capability status in docs.
|
|
80
|
+
|
|
81
|
+
## 5. Evaluation methodology — evidence hierarchy
|
|
82
|
+
|
|
83
|
+
Prioritize (highest to lowest):
|
|
84
|
+
1. Objective postcondition inspection in an isolated test workspace and independently executed acceptance tests.
|
|
85
|
+
2. Deterministic validators over attributable artifacts and tool traces.
|
|
86
|
+
3. Human-reviewed gold labels with reviewer agreement/adjudication.
|
|
87
|
+
4. Calibrated model-graded rubrics with judge/model/version, prompt and reference logged.
|
|
88
|
+
5. Unverified heuristics and language-only similarity: useful as hints, **never as proof of task completion**.
|
|
89
|
+
|
|
90
|
+
Implement case-level statuses `pass`, `fail`, `inconclusive`, `error`, `skipped`, `not_supported` with machine-readable reasons; retain whether observations were collected by EvalSutra or externally supplied. A missing trace is `inconclusive`, not a pass. Separate **coverage** from **quality**. Provide weighted success, confidence/uncertainty, critical failure counts and safety-gate overrides without score inflation. Avoid one opaque universal 0–100 metric. Make scoring formulas explicit and versioned.
|
|
91
|
+
|
|
92
|
+
Core grading types: exact, normalized exact, presence/absence, regex with safety constraints, JSON schema, API contract, file tree expected state, test process exit, diff constraints, tool-required/forbidden/ordering, latency/budget and structured rubric. Each grader should specify inputs, supported languages, risk level and provenance. Unsafe assertions against user-controlled code run only in a sandbox and with permission.
|
|
93
|
+
|
|
94
|
+
## 6. Statistical rigor and fair comparisons
|
|
95
|
+
|
|
96
|
+
For stochastic models, record repeat count, parameters, seeds where supported, sample variance, confidence intervals, failures and timeouts. Use paired tasks for candidate comparison; avoid treating unequal fixtures, changed models, tool availability or budgets as controlled comparisons. Compute success@k / pass@k only if definitions, trials and assumptions are explicit; provide confidence bands when appropriate. Track false positives and false negatives of graders against a curated human-reviewed validation set.
|
|
97
|
+
|
|
98
|
+
Benchmark contamination controls: separate train/dev/test sets, immutable hidden tests where warranted, fixture provenance and licensing, no inclusion of private keys/PII. Never encourage agents to train on or copy private test solutions. Prevent agent from editing benchmark tests during evaluation. Support versions and migration notes.
|
|
99
|
+
|
|
100
|
+
## 7. Security, privacy, trust and sandboxing
|
|
101
|
+
|
|
102
|
+
Principle: **no default remote execution and no hidden network access**. Treat all agent outputs, repository files, tool outputs, URLs, and test fixtures as untrusted data. Prompt injection cannot elevate authority. For live code-executing agents require explicit repository consent, least-privilege isolated workspaces, non-root process/container or VM isolation where available, locked-down network by default, resource limits, executable/tool allowlists, synthetic credentials, protected secret roots and sanitized logs. Never execute downloaded skill scripts automatically.
|
|
103
|
+
|
|
104
|
+
Threat model: host compromise, filesystem escape, credential leakage, insecure MCP servers, malicious dependency/postinstall, tool output injection, destructive edits, supply chain, SSRF, untrusted report HTML, PII exposure, cost exhaustion, denial of service, evaluator gaming, grader leakage and compromised CI. Use no raw Markdown-to-HTML without escaping/sanitizing. Never publish real attack payload secrets or expose protected artifacts. Provide accessible disclosure policy, dependency pinning, SBOM plan, secret scanning, GitHub permissions and protected release procedures.
|
|
105
|
+
|
|
106
|
+
For server mode: authentication, RBAC, tenant isolation, per-user quotas, CSRF/CORS considerations, audit logs, encryption and retention policy, vault-backed credentials, safe job queues, explicit cost caps, rate limits and credential redaction. Choose an architecture that can be self-hosted; make hosted deployment optional and cost-accountable.
|
|
107
|
+
|
|
108
|
+
## 8. Architecture at all levels
|
|
109
|
+
|
|
110
|
+
Produce and implement an architecture that covers enterprise context, product/solution architecture, system context/C4, container/component diagrams, runtime and dataflow architecture, deployment topology, threat model, HLD/LLD, API boundaries, storage schemas, message contracts, failure modes, observability, CI/CD, extension points and ADRs.
|
|
111
|
+
|
|
112
|
+
Recommended design:
|
|
113
|
+
- Portable TypeScript/Node.js CLI and core for three OSes, Node >=20 (or document changes).
|
|
114
|
+
- Deterministic evaluator and typed schema validators free of AI-service dependencies.
|
|
115
|
+
- Event-sourced or append-only evaluation run record with manifest checksums.
|
|
116
|
+
- Runner interface: replay, manually recorded, IDE-native, local, API and isolated process.
|
|
117
|
+
- Pluggable scorer contract; isolated evaluator adapters; deterministic graded evidence.
|
|
118
|
+
- Report engine with JSON/Markdown/HTML, optional SARIF/JUnit exports.
|
|
119
|
+
- Model/agent/IDE adapter registry with explicit capability flags.
|
|
120
|
+
- Optional browser UI/visualization and API server in separate packages.
|
|
121
|
+
- Optional Python adapter for Inspect AI/DeepEval without forcing Python onto all users.
|
|
122
|
+
|
|
123
|
+
Use sensible monorepo packaging and avoid uncontrolled complexity. Start with a working vertical slice. Use database only if persistence needs justify it; local files and SQLite may be sufficient initially. Preserve easy one-command local start, full test automation and an extensible schema.
|
|
124
|
+
|
|
125
|
+
## 9. Canonical folder / repository layout
|
|
126
|
+
|
|
127
|
+
Create a single repository root `evalsutra/` containing:
|
|
128
|
+
|
|
129
|
+
```text
|
|
130
|
+
evalsutra/
|
|
131
|
+
├── README.md, AGENTS.md, CLAUDE.md, GEMINI.md, LICENSE
|
|
132
|
+
├── SECURITY.md, CONTRIBUTING.md, CODE_OF_CONDUCT.md
|
|
133
|
+
├── GOVERNANCE.md, ROADMAP.md, CHANGELOG.md
|
|
134
|
+
├── package.json, tsconfig.json (if applicable), lockfile
|
|
135
|
+
├── .github/workflows/ (lint, test, security, build, release)
|
|
136
|
+
├── .claude-plugin/plugin.json (when plugin is publishable)
|
|
137
|
+
├── .agents/skills/ (canonical portable agent skills)
|
|
138
|
+
├── adapters/ (claude-code, cursor, codex, gemini-cli, copilot, generic, server, local-model)
|
|
139
|
+
├── packages/
|
|
140
|
+
│ ├── cli/ # doctor, init, run, report, compare, install, validate
|
|
141
|
+
│ ├── core/ # canonical schemas and orchestration contracts
|
|
142
|
+
│ ├── dataset/ # fixtures, integrity, sampling
|
|
143
|
+
│ ├── runner/ # controlled execution and replay
|
|
144
|
+
│ ├── judges/ # deterministic, human, optional model graders
|
|
145
|
+
│ ├── trace/ # OTEL and normalizer
|
|
146
|
+
│ ├── gates/ # evidence and release confidence policies
|
|
147
|
+
│ ├── storage/ # local artifacts and optional DB
|
|
148
|
+
│ ├── reporters/ # JSON, MD, HTML, SARIF, CSV
|
|
149
|
+
│ ├── dashboard/ # optional UI
|
|
150
|
+
│ ├── api/ # optional self-hosted server
|
|
151
|
+
│ └── integrations/ # Promptfoo, Inspect, frameworks, providers
|
|
152
|
+
├── suites/ # canonical eval suite definitions
|
|
153
|
+
├── examples/ # local zero-key working examples
|
|
154
|
+
├── fixtures/ # synthetic and licensed cases only
|
|
155
|
+
├── schemas/ # JSON schemas for versioned APIs
|
|
156
|
+
├── policies/ # gate profiles, permissions and budgets
|
|
157
|
+
├── templates/ # issue, report, scorecard, ADR
|
|
158
|
+
├── workflows/ # manual/CI/perf/portability eval workflows
|
|
159
|
+
├── scripts/ # .ps1 .cmd .sh cross-platform shims
|
|
160
|
+
├── tests/ # unit, contract, integration, OS, E2E, security
|
|
161
|
+
└── docs/ # architecture, plugin inventory, status, API, benchmarks
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
Don't add empty package directories and claim they are implemented. At each milestone, add only necessary files and keep a feature-status document. Ensure commands really execute after fresh clone.
|
|
165
|
+
|
|
166
|
+
## 10. Required CLI and public API
|
|
167
|
+
|
|
168
|
+
Design and incrementally implement: `evalsutra doctor`, `init`, `validate`, `install`, `list-skills`, `run`, `compare`, `report`, `gate`, `matrix`, `audit-plugin`, `import-traces`, and `serve` (optional). Every command must have `--help`, schema validation, typed errors, meaningful exit codes, JSON output and no surprise network calls. Support `--dry-run`, `--offline`, `--max-cost`, `--timeout`, `--seed`, `--repeat`, `--parallel` where relevant.
|
|
169
|
+
|
|
170
|
+
Sample evaluation:
|
|
171
|
+
```
|
|
172
|
+
evalsutra run --suite suites/demo.json --runner replay --observations examples/observations.json --out artifacts/demo
|
|
173
|
+
```
|
|
174
|
+
Output manifest: suite ID, dataset digest, agent+model+host descriptor if known, graded assertions, method/source, evidence references, timestamps, failures, policy decision and recommended review actions.
|
|
175
|
+
|
|
176
|
+
Quality gate semantics: `NO_GO`, `REVIEW`, `GO_CANDIDATE`; candidate never means actual deployment approval. On unsupported/missing evidence, don't silently produce a passing badge.
|
|
177
|
+
|
|
178
|
+
## 11. Evaluation workflows and benchmarks
|
|
179
|
+
|
|
180
|
+
Include representative synthetic and reproducible suites for:
|
|
181
|
+
- factual correctness and grounded answers
|
|
182
|
+
- coding tasks with independent compile/test/postconditions
|
|
183
|
+
- tool-use authorization/ordering and least privilege
|
|
184
|
+
- JSON/function-calling format/contract compliance
|
|
185
|
+
- planning vs actual task completion and task artifacts
|
|
186
|
+
- prompt-injection resilience using synthetic planted instructions
|
|
187
|
+
- data leakage checks using synthetic canary values
|
|
188
|
+
- long horizon multi-step workflow and recovery from failures
|
|
189
|
+
- timeouts, retries and rate-limit resilience
|
|
190
|
+
- multi-agent delegation and handoff correctness
|
|
191
|
+
- retrieval and citation groundedness
|
|
192
|
+
- IDE-specific support and runtime features
|
|
193
|
+
- cost, latency, throughput and environment constraints
|
|
194
|
+
- cross-language and cross-platform edge cases
|
|
195
|
+
- accessibility and UI task end-state verification (optional browser)
|
|
196
|
+
|
|
197
|
+
Offer fixtures as portable JSON/YAML with JSON Schema validation, clearly mark data provenance/licensing and whether benchmark is public/hidden. Provide working self-test suites for graders so the framework can evaluate its own reliability.
|
|
198
|
+
|
|
199
|
+
## 12. Integrations — prefer genuine adapters over rebrands
|
|
200
|
+
|
|
201
|
+
Review existing frameworks and implement optional adapters (not wholesale clones):
|
|
202
|
+
- **Promptfoo:** import/export suites, assertions, results, traces and optional red-team tasks.
|
|
203
|
+
- **Inspect AI:** mapping between EvalSutra runs and Inspect tasks/logs, optional Python package.
|
|
204
|
+
- **OpenTelemetry GenAI:** trace correlation, tool spans, duration, model metadata where available.
|
|
205
|
+
- **OpenAI agent traces/evals:** official SDK/API where authorized, preserving provenance.
|
|
206
|
+
- **LangSmith, Langfuse, MLflow, DeepEval:** optional integrations only after user demand.
|
|
207
|
+
- **Playwright:** browser task end-state and screenshots; isolated local test sites.
|
|
208
|
+
- **GitHub Actions:** clean OS matrix, report artifacts, conservative gated checks; no unnecessary secrets.
|
|
209
|
+
- **Ollama:** optional local judge/agent provider when installed.
|
|
210
|
+
|
|
211
|
+
For every integration, check current official docs, licenses, version support and auth. Provide offline mock tests to validate contracts without real external accounts.
|
|
212
|
+
|
|
213
|
+
## 13. Plugin/skill/MCP discovery and installation policy
|
|
214
|
+
|
|
215
|
+
**Inspect tools already installed before recommending additions.** Discover official plugin marketplaces, installed extensions, CLI version and available capabilities. Build `docs/PLUGIN_AND_SKILL_CATALOG.md` with verified install sources and specific reasons. Suggest these categories for development:
|
|
216
|
+
|
|
217
|
+
Priority 1: Claude Code Setup, Superpowers, Feature Dev, Code Review, Security Guidance, GitHub, Context7.
|
|
218
|
+
Priority 2: Playwright, TypeScript LSP, Serena, PR Review Toolkit, CLAUDE.md Management.
|
|
219
|
+
AI eval skills: Promptfoo evals, provider setup and red-team setup/run (red team only against authorized local targets).
|
|
220
|
+
Optional: Code Simplifier, Commit Commands, Docker/containers, Inspect AI, OpenTelemetry, local Ollama.
|
|
221
|
+
|
|
222
|
+
**Never globally install everything automatically.** Prefer project-scoped additions, show package/plugin identity, trust source, version, required permissions, disk/cost/network effects, installation command and uninstall steps. Get explicit user permission before changing global Claude settings, installing executable MCP servers, editing home directories, linking third-party accounts, downloading models, or provisioning cloud services. Don't invent plugin identifiers or claim an install succeeded without verification. Never install redundant/broken integrations. Keep EvalSutra core working with none of them.
|
|
223
|
+
|
|
224
|
+
If Claude plugin packaging is supported, create `.claude-plugin/plugin.json`, package `skills/` and optional agents/hooks, and test package discovery. If Cursor/GitHub plugins support compatible standards, create valid separate manifests, not one universal incompatible manifest. Do not change model auth settings.
|
|
225
|
+
|
|
226
|
+
## 14. CI/CD and supply chain
|
|
227
|
+
|
|
228
|
+
GitHub Actions matrix: Linux, Windows, macOS, Node LTS versions. Offline tests required for every PR. Keep network and provider APIs out of mandatory PR checks. Add typecheck/lint/test, dependency review, secret scan, threat tests, schema validation, reproducible sample run, packaging checks, documentation links and license checks. Use least privilege and pinned third-party workflow versions (prefer commit SHAs for release readiness). Require reviewed PRs and public responsible disclosure policy. Separate integration tests needing credentials, labels and opt-in budgets. Publish no false CI/pass/security badges.
|
|
229
|
+
|
|
230
|
+
## 15. Dashboard UX and report design
|
|
231
|
+
|
|
232
|
+
If implementing an optional UI, create an accessible design system: dark/light themes, mobile-responsive, keyboard navigation, WCAG 2.2 AA target, proper semantic structure, color not the only severity signal. Views: dataset explorer, run details, trace timeline, scoring breakdown, reproducibility/provenance, comparison matrix, regressions, cost/latency trends, audit trail, cross-IDE compatibility matrix, report exports. Charts must label units, sample counts, uncertainty and missing data. Never fabricate analytics or claim unrun evaluations. Ensure default privacy: no uploads without consent.
|
|
233
|
+
|
|
234
|
+
## 16. Verification and self-evaluation
|
|
235
|
+
|
|
236
|
+
The platform must test itself. Include examples intentionally designed to pass, fail, and be inconclusive; test calibration, ranking stability, tool ordering, missing evidence, malformed artifacts, injection content, cost/timeout boundaries and false score inflation. For every module: unit tests, contract tests, integration tests where feasible, negative tests, deterministic fixture, code-quality checks and documentation. Report test runner/platform/version, actual test count, failures and unsupported checks. Do not claim OS support passed unless corresponding CI actually executed.
|
|
237
|
+
|
|
238
|
+
## 17. Release acceptance gates
|
|
239
|
+
|
|
240
|
+
The first publishable v0.x release must satisfy all of these:
|
|
241
|
+
- Fresh clone with documented supported Node version can run core offline and CI tests.
|
|
242
|
+
- Portable `SKILL.md` has valid metadata and is usable in at least one verified host.
|
|
243
|
+
- Local `doctor`, suite validator, recorded run evaluator, compare/baseline and report work.
|
|
244
|
+
- Evidence hierarchy and inconclusive statuses are correct, especially on missing data.
|
|
245
|
+
- A synthetic guarded test fails if a forbidden tool is recorded.
|
|
246
|
+
- At least one safe CLI skill installer works without touching global scope.
|
|
247
|
+
- Docs explicitly state what is implemented vs planned, and how to contribute.
|
|
248
|
+
- Repo includes Apache-2.0, README, SECURITY, CONTRIBUTING, CODE_OF_CONDUCT, CHANGELOG, ROADMAP.
|
|
249
|
+
- No baked-in credentials, no unreviewed remote code execution, no proprietary dependencies in offline core.
|
|
250
|
+
- Real CI results, not made-up pass badges. Manual review required for release.
|
|
251
|
+
|
|
252
|
+
For later releases: isolated live runner, multiple verified host adapters, audit-plugin, OpenTelemetry, Promptfoo/Inspect, optional dashboard and secure hosted API. Add only when tests can validate them.
|
|
253
|
+
|
|
254
|
+
## 18. Development delivery phases
|
|
255
|
+
|
|
256
|
+
P0 — Discover existing starter; validate toolchain; agree security boundaries; produce capability matrix and prioritized gap list.
|
|
257
|
+
P1 — Robust offline evaluator: schema validation, import/normalize, evidence labels, truthful scoring, gate semantics, report and CLI unit tests.
|
|
258
|
+
P2 — Regression harness, reproducible hashes/seeds, statistical summaries and quality-calibration datasets.
|
|
259
|
+
P3 — IDE portability: install, detect and validate native skills, where supported live runner, precise compatibility status and docs.
|
|
260
|
+
P4 — Sandbox and controlled local execution; safe command allowlist and consent workflows.
|
|
261
|
+
P5 — Integrations: Promptfoo, Inspect AI, OTEL, local models and optionally provider APIs.
|
|
262
|
+
P6 — Multi-agent coordination, server/API with auth, secure optional dashboard, stronger threat evaluation.
|
|
263
|
+
P7 — Release hardening, community, tutorials, governed benchmarks and public compatibility lab.
|
|
264
|
+
|
|
265
|
+
At the end of each phase publish a factual report: implemented files, changes made, commands executed, pass/fail evidence, security caveats, remaining unsupported capabilities, cost implications, ADRs and next priorities.
|
|
266
|
+
|
|
267
|
+
## 19. Contributor and reputation strategy
|
|
268
|
+
|
|
269
|
+
Public README should explain why agent evals matter, show a **genuinely working** 30-second no-key example, outputs, screenshots only if captured, feature-status matrix and docs. Contributor guide: easy-first issues, fixture proposals, grader plugin API, adapter testing, evidence standards, review checklist, attribution/licensing, governance, conduct and vulnerability reporting. Provide benchmark contribution rules that prohibit real secrets and unlicensed evaluation data. Use semver and changelog. Publish transparent limitations and reproducibility metadata.
|
|
270
|
+
|
|
271
|
+
Branding: `EvalSutra AI` and `EvalSutra Prime`; title `The Agent Evaluation & Reliability Intelligence Platform`; tagline `Test Intelligence. Measure Truth. Build Trust.`; `Created by Vatsal Shah` in project metadata and main README. Proposed repository `evalsutra`, command `evalsutra`, package `@vatsalshah/evalsutra` only after confirming ownership/availability. Name/trademark clearance must be performed before publishing.
|
|
272
|
+
|
|
273
|
+
## 20. Immediate action instructions for Claude
|
|
274
|
+
|
|
275
|
+
1. Inspect existing root. Enumerate implemented functionality by *executing* tests and CLI samples.
|
|
276
|
+
2. Read these instructions as goals, not as proof that all phases exist.
|
|
277
|
+
3. Make a clear architecture + capability gap analysis; choose a narrow high-value milestone.
|
|
278
|
+
4. Implement functioning code, tests, docs and error handling.
|
|
279
|
+
5. Show accurate command output and test counts; clearly identify Linux/Windows/macOS results separately.
|
|
280
|
+
6. Enumerate optional plugins already present; recommend only verified ones; ask before any global install.
|
|
281
|
+
7. Never store actual AI provider keys in source, logs, examples or reports.
|
|
282
|
+
8. Follow secure-by-default, offline-first, evidence-first policies and limit live target operations to authorized isolated environments.
|
|
283
|
+
9. Continue implementation through phases as far as the environment safely allows. If constrained, finish the currently functioning vertical slice and provide a factual backlog; do not claim full completion.
|
|
284
|
+
|
|
285
|
+
**Do not stop after a plan. Build, test and document working code in the single `evalsutra/` root folder.**
|
|
286
|
+
|
|
287
|
+
---
|
|
288
|
+
|
|
289
|
+
# APPENDIX A — NOVEMBER 2026 ADVANCED TOOLING & INTEGRATION PLAN
|
|
290
|
+
|
|
291
|
+
# EvalSutra AI — Advanced Tooling, Differentiation & Integration Roadmap
|
|
292
|
+
|
|
293
|
+
**Project:** EvalSutra AI — Agent Evaluation & Reliability Intelligence Platform
|
|
294
|
+
**Creator:** Vatsal Shah
|
|
295
|
+
**Proposed namespace:** `evalsutra`
|
|
296
|
+
**Status:** Engineering plan. External connectors and advanced modules below are NOT implemented in the starter.
|
|
297
|
+
**Updated:** 2026-11-01
|
|
298
|
+
|
|
299
|
+
## 0. Guiding product contract
|
|
300
|
+
|
|
301
|
+
The most valuable objective is an **agent reliability laboratory** rather than a wrapper around commercial evaluators. EvalSutra must measure demonstrable changes to an environment, tool correctness, repeatability, security, privacy, cost and cross-host behavior, using a **reproducible evidence record**.
|
|
302
|
+
|
|
303
|
+
**Non-negotiable:** The default CLI must work on Windows/macOS/Linux without model API keys, cloud account, remote service, container engine, Python runtime, or paid subscription. Offline JSON fixture evaluation, deterministic assertions, score comparisons and reports must keep working. Real agent execution, browser testing, cloud APIs, model judges and sandbox orchestration must be optional and capability-detected. An IDE subscription/session cannot be scraped or reused as a hidden API; use only documented host integrations.
|
|
304
|
+
|
|
305
|
+
**Disclosure:** A plugin, an SDK, a framework adapter, an optional locally hosted service and a CLI test runner are distinct things. Do not call every integration a 'Claude plugin'. List and verify actual compatibility, credentials, license, cost and permissions before installing anything.
|
|
306
|
+
|
|
307
|
+
## 1. Additional tools beyond the previous catalog (priority matrix)
|
|
308
|
+
|
|
309
|
+
| Integration | Priority | What it contributes | Default-mode dependency? | Notes |
|
|
310
|
+
|---|---|---|---|---|
|
|
311
|
+
| **Harbor + Terminal-Bench** | P1 | Realistic containerized coding and terminal-agent benchmark harness | No | Official https://github.com/harbor-framework/harbor ; Docker/runtime and model authentication may be required for live runs. |
|
|
312
|
+
| **MCP Inspector** | P1 | Test MCP server tools, resources, prompts, schemas and error handling; CLI automation | No | Official https://github.com/modelcontextprotocol/inspector ; test only trusted / isolated MCP servers. |
|
|
313
|
+
| **OpenTelemetry GenAI conventions / OTLP** | P1 | Portable agent traces, spans, events, tokens and tool calls | No | https://opentelemetry.io/docs/specs/semconv/ ; GenAI conventions evolve; version the mapping. |
|
|
314
|
+
| **Opik** | P2 (choose one observability bridge first) | Self-hostable / remote trace & eval experiments, optional MCP integration | No | https://www.comet.com/docs/opik/ ; assess deployment, licensing and data handling. |
|
|
315
|
+
| **Arize Phoenix** | P2 alternative to Opik | Local agent trace inspection, datasets, experiments | No | https://arize.com/phoenix/ ; source-available ELv2, **not Apache-2.0**; do not bundle code without review. |
|
|
316
|
+
| **Microsoft PyRIT** | P2 | Authorized red-team and prompt-injection robustness scenarios | No | https://microsoft.github.io/PyRIT/ ; safe, controlled targets only. |
|
|
317
|
+
| **Hypothesis (Python) / fast-check (TypeScript)** | P1 | Property-based test generation of graders, parsers and adapters | Dev-only | https://hypothesis.readthedocs.io/ and https://fast-check.dev/ . |
|
|
318
|
+
| **uv** | P2 | Efficient optional Python tool and benchmark environment management | No | https://docs.astral.sh/uv/ ; never force Python on JS-only users. |
|
|
319
|
+
| **DuckDB / SQLite** | P1 optional storage backend | Local analytical runs/datasets, efficient aggregate comparisons | No | Use JSON flat-file fallback; SQLite for metadata, DuckDB for analytics only when justified. |
|
|
320
|
+
| **Gitleaks** | P1 CI | Detect accidentally committed secrets | No | https://github.com/gitleaks/gitleaks . |
|
|
321
|
+
| **OSV-Scanner** | P1 CI | Dependency vulnerability checking with actionable evidence | No | https://github.com/google/osv-scanner . |
|
|
322
|
+
| **OpenSSF Scorecard** | P2 CI | Public-repository security posture checks | No | https://github.com/ossf/scorecard . |
|
|
323
|
+
| **npm trusted publishing + provenance** | P2 publish | Safer official package publication via OIDC | No | https://docs.npmjs.com/trusted-publishers/ . |
|
|
324
|
+
| **Changesets or Release Please** | P2 publish, choose one | Versioning, changelogs, automated release PRs | No | Avoid adopting both. |
|
|
325
|
+
| **DVC** | P3 optional large-dataset workflows | Version large benchmark datasets without committing sensitive blobs | No | https://dvc.org/ ; prefer simple content hashes and Git for small suites. |
|
|
326
|
+
| **Ragas** | P3 specialized RAG evaluation | Retrieval-augmented agent grounding tests | No | https://docs.ragas.io/ ; activate only for RAG suites. |
|
|
327
|
+
|
|
328
|
+
**Already recommended; do not count as new or install twice:** Promptfoo, Inspect AI, Docker/Podman, Playwright, Ollama, GitHub Actions, Claude Code Setup, Superpowers, Code Review, Feature Dev, Security Guidance, GitHub, Context7, Serena, TypeScript LSP, PR Review Toolkit, Langfuse, LangSmith, MLflow and DeepEval. Treat the latter hosted or heavier frameworks as possible adapters, not required dependencies.
|
|
329
|
+
|
|
330
|
+
### Tool decisions (avoid overengineering)
|
|
331
|
+
|
|
332
|
+
1. Make **Harbor** the first live *coding-agent benchmark* adapter; keep existing deterministic replay as the default.
|
|
333
|
+
2. Make **MCP Inspector** the first *protocol/tool testing* integration; define stable imported-result JSON and a CLI wrapper if supported.
|
|
334
|
+
3. Implement **OpenTelemetry import/export** as the shared telemetry interchange; only **one** optional observability UI initially (choose Opik or Phoenix after proof-of-concept).
|
|
335
|
+
4. Add **PyRIT** after access controls and isolated fixtures; use it for authorized safety tests, not unsupervised internet targets.
|
|
336
|
+
5. Add **fast-check** (and Hypothesis only for the Python adapter) for parser and score-engine fuzz testing.
|
|
337
|
+
6. Add **Gitleaks, OSV-Scanner, CodeQL** (where license/config allow), pinned CI actions and package provenance to public-release hardening.
|
|
338
|
+
7. Do **not** force a heavy web dashboard or full cloud stack into the first milestone. CLI and HTML reports first; dashboard later.
|
|
339
|
+
|
|
340
|
+
## 2. Distinctive product innovations to build
|
|
341
|
+
|
|
342
|
+
These are proposed proprietary-to-EvalSutra *features* in the sense of product differentiation, not claims of patentability or novel scientific algorithms.
|
|
343
|
+
|
|
344
|
+
**A. EvalSutra Repro Capsule (`repro`)**
|
|
345
|
+
A portable directory or archive containing suite definition and schema version; agent and adapter metadata; sanitized input and output; tool trace; redacted environment/OS/runtime fingerprints; execution timestamps; sanitized artifacts; hashes; and deterministic scoring configuration. A consumer can replay *grading* without rerunning an AI model. No secrets or credentials should be included. Use a manifest and SHA-256 digest per item; do not claim tamper-proofing unless signed and verified.
|
|
346
|
+
|
|
347
|
+
**B. EvalSutra Compat Doctor (`compat`)**
|
|
348
|
+
Host capability probes for Claude Code, Cursor, Codex, Gemini CLI, Copilot and generic CLI/MCP. Report only **verified**, **partially verified**, **unsupported**, or **unknown** per capability: skill discovery, agent invocation, tools, filesystem edits, traces, end-state verification, background execution, cost data and network policy. Do not say 'all IDEs supported' without tested contracts. Host-specific official paths must be version-checked.
|
|
349
|
+
|
|
350
|
+
**C. EvalSutra Trace Lens (`trace`)**
|
|
351
|
+
Trace parser that highlights unauthorized actions, skipped checks, tool-order errors, hallucinated success claims, redundant tool loops, timeouts, fallback behavior and tool-call failures. Provide direct evidence IDs and machine-readable discrepancy codes.
|
|
352
|
+
|
|
353
|
+
**D. EvalSutra Judge Calibration Lab (`calibrate`)**
|
|
354
|
+
Optional AI judges are tested against human-reviewed gold labels. Track label agreement, per-category disagreement, prompt/model versions, rubrics, class balance and judge cost. AI judge output is an opinion, **not** objective ground truth. No-key deterministic scoring must remain independent.
|
|
355
|
+
|
|
356
|
+
**E. EvalSutra Agent Scorecard (`scorecard`)**
|
|
357
|
+
A versioned scorecard that separates objective task completion, tool compliance, safety boundaries, reproducibility, latency and cost. Show missing data rather than treating missing observations as zero or success. Publish only accompanied by sample size, task categories, environment and confidence intervals when applicable.
|
|
358
|
+
|
|
359
|
+
**F. EvalSutra Regression Firewall (`gate`)**
|
|
360
|
+
Compare candidate and baseline on matched, version-pinned tasks. Severe confirmed policy violations or benchmark invalidity block automatically. For other scores, use explicit thresholds with uncertainty; abstain when insufficient evidence. Produce PASS, FAIL, or INCONCLUSIVE—never fabricate certainty.
|
|
361
|
+
|
|
362
|
+
**G. EvalSutra Benchmark Integrity (`integrity`)**
|
|
363
|
+
Detect fixture duplication, leaked answers, test contamination risks, dataset licensing ambiguity, path leakage, benchmark drift and suspicious harness tampering. Reserve blind holdout datasets, with sealed expectations stored separately from the agent's input files.
|
|
364
|
+
|
|
365
|
+
**H. EvalSutra Failure-to-Fix Loop (`triage`)**
|
|
366
|
+
Cluster failures by mechanism and produce evidence-backed issue drafts, reproduction commands and proposed fixes. Never rewrite agent code or submit PRs without explicit authorization. Verify any proposed change using a fresh, isolated rerun.
|
|
367
|
+
|
|
368
|
+
## 3. Multi-level platform architecture
|
|
369
|
+
|
|
370
|
+
**Level 1 — Product/enterprise:** local CLI; independent packages; opt-in hosted enterprise add-on; open schema and exporter; cross-project compatibility standards.
|
|
371
|
+
|
|
372
|
+
**Level 2 — Solution:** `CLI / IDE Skill / Server API` -> `Capability Discovery` -> `Suite Registry` -> `Execution Adapter` -> `Isolated Runner` -> `Evidence Collector` -> `Deterministic Evaluators` -> `Optional AI Judge` -> `Comparison/Gating` -> `Reports/Artifacts`. Runs must have stable IDs and redacted logs. A failed or interrupted runner must preserve clear provenance.
|
|
373
|
+
|
|
374
|
+
**Level 3 — Component/module contracts:**
|
|
375
|
+
|
|
376
|
+
- `core/schema`: versioned JSON Schema validated at ingress; suite, fixture, observation, trace, result, policy and manifest.
|
|
377
|
+
- `core/engine`: pure deterministic evaluators with typed results and explicit inconclusive reasons.
|
|
378
|
+
- `core/statistics`: paired comparisons, confidence intervals, nondeterminism and stratified reports.
|
|
379
|
+
- `core/security`: redaction, permissions, secrets hygiene, provenance, size/time/network budgets.
|
|
380
|
+
- `runner`: read-only replay by default; opt-in sandbox execution with CPU/memory/time quotas, temporary workspace and outbound network denied by default.
|
|
381
|
+
- `adapters/host`: capability probes + execution contracts per IDE, never private token scraping.
|
|
382
|
+
- `integrations/harbor`, `/mcp-inspector`, `/otel`, `/opik`, `/phoenix`, `/pyrit` feature-flagged.
|
|
383
|
+
- `storage`: local JSON append-only artifacts; opt-in SQLite/DuckDB; no server required.
|
|
384
|
+
- `reports`: JSON, Markdown, HTML, JUnit, SARIF when meaningful, and optional OTLP export.
|
|
385
|
+
- `server`: optional API behind authenticated HTTPS, multi-tenant isolation, RBAC, scoped tokens, quotas, audit logs and queues.
|
|
386
|
+
- `dashboard`: optional accessible browser UI for task coverage, traces, comparisons, and release gates.
|
|
387
|
+
|
|
388
|
+
**Level 4 — Code-level:** Testable pure functions; time-independent deterministic score formulas; no `eval` or arbitrary commands from dataset definitions; validated input; size limits; abortable workers; deterministic fixture sorting and stable hash schemas.
|
|
389
|
+
|
|
390
|
+
## 4. New file/folder layout to add (preserve the working starter)
|
|
391
|
+
|
|
392
|
+
```text
|
|
393
|
+
evalsutra/
|
|
394
|
+
docs/
|
|
395
|
+
ADVANCED_TOOLING_ROADMAP.md
|
|
396
|
+
EVALUATION_METHODOLOGY.md # implement when relevant
|
|
397
|
+
BENCHMARK_INTEGRITY.md # implement when relevant
|
|
398
|
+
packages/
|
|
399
|
+
core/metrics/ # new tested metrics
|
|
400
|
+
core/manifest/ # repro capsule schema
|
|
401
|
+
core/trace/ # trace rules
|
|
402
|
+
core/compat/ # capability matrix
|
|
403
|
+
core/judge/ # optional and calibrated
|
|
404
|
+
runner/sandbox/ # optional verified isolation
|
|
405
|
+
statistics/ # comparative evaluation
|
|
406
|
+
reporters/ # append to existing reporter
|
|
407
|
+
server/ # optional: no placeholder API claims
|
|
408
|
+
dashboard/ # optional: later
|
|
409
|
+
integrations/
|
|
410
|
+
harbor/
|
|
411
|
+
mcp-inspector/
|
|
412
|
+
opentelemetry/
|
|
413
|
+
opik/ # initial observability choice
|
|
414
|
+
phoenix/ # alternative; keep off by default
|
|
415
|
+
pyrit/
|
|
416
|
+
suites/
|
|
417
|
+
fixtures/
|
|
418
|
+
gold/
|
|
419
|
+
adversarial/
|
|
420
|
+
compatibility/
|
|
421
|
+
schemas/
|
|
422
|
+
repro-manifest.schema.json
|
|
423
|
+
capability-report.schema.json
|
|
424
|
+
trace-event.schema.json
|
|
425
|
+
tests/
|
|
426
|
+
contract/
|
|
427
|
+
property/
|
|
428
|
+
security/
|
|
429
|
+
regression/
|
|
430
|
+
.github/
|
|
431
|
+
workflows/
|
|
432
|
+
security.yml
|
|
433
|
+
publish.yml # enable only with trusted publisher
|
|
434
|
+
```
|
|
435
|
+
|
|
436
|
+
Create folders only when implementing actual files and testable features. Empty scaffolds must not be described as implementation. Keep `npm test`, `npm run doctor` and `npm run demo` working during migration.
|
|
437
|
+
|
|
438
|
+
## 5. Phased build roadmap with objective gates
|
|
439
|
+
|
|
440
|
+
| Phase | Primary deliverable | Exit/acceptance evidence |
|
|
441
|
+
|---|---|---|
|
|
442
|
+
| **P0: Baseline** | Inventory starter, CI, threat model and full test pass | Existing tests green; present feature-status matrix; no regressions. |
|
|
443
|
+
| **P1: Portable evidence** | Repro Capsule, schema validation, trace-event normalization and redaction | Offline golden fixture survives export/import/replay; secrets never appear in output. |
|
|
444
|
+
| **P2: Trustworthy grading** | Repeatability, missing-data handling, paired comparisons and gate policies | Tests cover PASS/FAIL/INCONCLUSIVE; no AI key required; reproducible scores. |
|
|
445
|
+
| **P3: Protocol compatibility** | MCP Inspector adapter and host Compat Doctor | Feature matrix based on real smoke tests; unsupported capabilities reported honestly. |
|
|
446
|
+
| **P4: Real-world evaluation** | Harbor adapter + isolated runner | Known sample benchmark executed in approved sandbox; outcomes and sandbox telemetry captured; missing runtime returns actionable 'unsupported'. |
|
|
447
|
+
| **P5: Trace observability** | OpenTelemetry normalization, optional Opik/Phoenix adapter | Import + export round trip preserves event provenance; redaction verified. |
|
|
448
|
+
| **P6: Controlled AI judges** | Local Ollama or host-supported judge integration, calibration dataset | Human-reviewed gold comparison and judge uncertainty reporting; no key required for deterministic mode. |
|
|
449
|
+
| **P7: Safety & integrity** | PyRIT bridge, contamination checks, permission tests | Authorized sandbox-only adversarial tasks; no unauthorized network operations. |
|
|
450
|
+
| **P8: CI, server & dashboard** | Public workflows, optional secure API, accessible UI, release pipeline | OS matrix checks; threat-model and auth tests; artifact provenance; dashboard works without disclosing sensitive data. |
|
|
451
|
+
| **P9: Community release** | Contributing guide, plugin contracts, sample datasets, versioned documentation | Clean-clone install, examples, transparent support matrix, release assets and SBOM. |
|
|
452
|
+
|
|
453
|
+
**Recommended first public milestone:** Phases P0–P3 plus one small Harbor proof of concept. Defer costly live benchmarks and multi-tenant hosting until maintainers can sustain them.
|
|
454
|
+
|
|
455
|
+
## 6. Benchmark and report design
|
|
456
|
+
|
|
457
|
+
### Evaluation case structure
|
|
458
|
+
|
|
459
|
+
- `case_id`, `suite_version`, `input_ref`, `target_agent`, `host`, `platform`, `expected_end_state` (sealed as needed), `allowed_tools`, `forbidden_tools`, `timeout_budget`, `network_policy`, `grader_versions`, `policy_ref`, `baseline_ref`.
|
|
460
|
+
- Every observation declares `source_type`: `actual_run`, `imported_trace`, `human_label`, `test_fixture`, `model_judge`, or `unverified_claim`.
|
|
461
|
+
- Every result declares `status`: `pass`, `fail`, `inconclusive`, `skipped`, or `error`, plus `evidence_refs`, `reason_codes`, `confidence/uncertainty` as applicable.
|
|
462
|
+
- Score aggregation must **not** silently count missing, skipped, unrun or inconclusive cases as passes.
|
|
463
|
+
|
|
464
|
+
### Multi-axis scorecard
|
|
465
|
+
|
|
466
|
+
Publish separate numbers for task success, tool correctness, authorization, robustness, latency, cost and repeatability. If presenting a composite score, show explicit weights and missing-data policy; default to no composite. Compare matched tasks, host versions, tool permissions and model modes; do not market rankings across incomparable setups.
|
|
467
|
+
|
|
468
|
+
### Redaction and user consent
|
|
469
|
+
|
|
470
|
+
Use synthetic data by default. Opt-in before uploading prompts, traces, screenshots, proprietary documents or code. Mask credentials and personal data *before* third-party export. Define retention controls and deletion in server mode. Disallow network egress and host mounts for untrusted benchmark agents unless explicitly approved. Do not mistake Docker alone for a sufficient trust boundary for malicious workloads; document isolation assumptions.
|
|
471
|
+
|
|
472
|
+
## 7. Tool review before installation
|
|
473
|
+
|
|
474
|
+
Claude must inventory actual installed plugins/MCP/CLI tools first. Evaluate publisher, maintenance, license, security advisories, version, data flow, telemetry, system requirements and cost. For each proposal state `required/optional`, `local/remote`, `free/paid potential`, `permission scope`, `installation command from verified docs`, `rollback command` and a minimal smoke test. Choose **project scope** by default. Never install globally or connect to external services without user permission. Do not run curl-to-shell installers blindly. Never store API keys in repository files. Use the user's IDE-native session only through published/authorized host capability.
|
|
475
|
+
|
|
476
|
+
## 8. New CLI proposals (specification, not implemented)
|
|
477
|
+
|
|
478
|
+
```sh
|
|
479
|
+
evalsutra repro export --run RUN_ID --out ./capsule
|
|
480
|
+
evalsutra repro verify ./capsule
|
|
481
|
+
evalsutra compat doctor --host claude-code
|
|
482
|
+
evalsutra trace analyze --input ./trace.json
|
|
483
|
+
evalsutra compare --baseline ./baseline --candidate ./candidate
|
|
484
|
+
evalsutra gate --policy ./policies/release.json
|
|
485
|
+
evalsutra benchmark run --adapter harbor --suite ./suites/coding --sandbox approved
|
|
486
|
+
evalsutra mcp inspect --config ./fixtures/mcp.json
|
|
487
|
+
evalsutra judge calibrate --gold ./suites/gold --provider local
|
|
488
|
+
evalsutra toolchain doctor
|
|
489
|
+
```
|
|
490
|
+
|
|
491
|
+
Commands are proposed; Claude must build them before documenting as working. Failure codes must be stable and meaningful; distinguish unavailable runtime, failed test, invalid evidence, and genuine security violation.
|
|
492
|
+
|
|
493
|
+
## 9. Suggested release quality gates
|
|
494
|
+
|
|
495
|
+
1. Deterministic core, CLI and format tests pass in a **Windows/macOS/Linux** CI matrix; absence of CI evidence means partial verification.
|
|
496
|
+
2. Replay succeeds without external credentials on a clean machine with the documented runtime installed.
|
|
497
|
+
3. Security scanning finds no unresolved **confirmed critical** defects; severity and triage processes documented.
|
|
498
|
+
4. Eval schemas and data migration paths are versioned and backward compatibility tested.
|
|
499
|
+
5. A malicious fixture cannot run shell commands merely by being parsed; untrusted agent code only executes via an approved isolation layer.
|
|
500
|
+
6. Judge evaluation never overrides a confirmed objective failure or prohibited action.
|
|
501
|
+
7. Reports state exactly which capabilities and benchmarks were run and expose the reproducibility capsule.
|
|
502
|
+
8. Starter examples include success, failure, inconclusive, timeout, missing evidence, tool mismatch and policy violation.
|
|
503
|
+
9. Public benchmarks disclose suite versions, sample sizes, environment, runtime, provider/model mode, cost collection method and uncertainty.
|
|
504
|
+
10. Licenses, SBOM, dependency checks, signed/provenance release options, security disclosure and contributor processes are documented.
|
|
505
|
+
|
|
506
|
+
## 10. Special rules for Claude implementation
|
|
507
|
+
|
|
508
|
+
- Read the existing master prompt **plus this document**; both apply. If scopes conflict, preserve safety and working deterministic core; record an ADR for design choices.
|
|
509
|
+
- Never claim plugin installation, IDE support, sandbox enforcement, model evaluation, server deployment or test execution until verified in the running environment.
|
|
510
|
+
- Use existing EvalSutra agents: Prime owns orchestration; Bench owns suites; Trace owns tools; Port owns compatibility; Guard owns safety; Judge owns calibration; Drift owns regression; Verify owns independent acceptance; Report owns evidence display. Do not create overlapping new agents simply to add features.
|
|
511
|
+
- Before writing large new modules, implement a thin vertical slice with fixtures, schema, CLI, tests and example output.
|
|
512
|
+
- Create evidence-rich PR-ready commits or patches; reserve network access and global changes for explicit authorization.
|
|
513
|
+
- In the final engineering report list: `completed`, `tested`, `not tested`, `planned`, `blocked`, `requires consent`, and the precise test commands and results.
|
|
514
|
+
|
|
515
|
+
## 11. Official references (re-verify before installation)
|
|
516
|
+
|
|
517
|
+
- Harbor: https://github.com/harbor-framework/harbor
|
|
518
|
+
- Terminal Bench: https://www.tbench.ai/
|
|
519
|
+
- Inspect AI eval registry: https://inspect.aisi.org.uk/evals/
|
|
520
|
+
- MCP Inspector: https://github.com/modelcontextprotocol/inspector
|
|
521
|
+
- OpenTelemetry: https://opentelemetry.io/docs/specs/semconv/
|
|
522
|
+
- Opik: https://www.comet.com/docs/opik/
|
|
523
|
+
- Phoenix: https://arize.com/phoenix/
|
|
524
|
+
- PyRIT: https://microsoft.github.io/PyRIT/
|
|
525
|
+
- Claude plugins: https://claude.com/marketplace/plugins
|
|
526
|
+
- npm trusted publishing: https://docs.npmjs.com/trusted-publishers/
|
|
527
|
+
- OpenSSF Scorecard: https://github.com/ossf/scorecard
|
|
528
|
+
- Gitleaks: https://github.com/gitleaks/gitleaks
|
|
529
|
+
- OSV-Scanner: https://github.com/google/osv-scanner
|
|
530
|
+
- GitHub Actions: https://docs.github.com/en/actions
|
|
531
|
+
|
|
532
|
+
---
|
|
533
|
+
|
|
534
|
+
**Publication discipline:** The starter is not a finished multi-host agent-evaluation platform. Until tests exist, all additions remain proposed. Keep marketing honest, developer trust high, and every benchmark verifiable.
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# 1. Executive summary
|
|
2
|
+
|
|
3
|
+
> **Test Intelligence. Measure Truth. Build Trust.** EvalSutra AI is a free, open-source, offline tool that tells you, with evidence, whether an AI agent did what it was supposed to do.
|
|
4
|
+
|
|
5
|
+
## What problem it solves
|
|
6
|
+
|
|
7
|
+
Teams ship AI agents and judge them by reading the agent's own words. An agent that says "Done, all tests pass" may have run no tests. Prose is not proof. EvalSutra grades **recorded evidence** (final output, tool calls, timing) against rules you write in advance, and refuses to guess when evidence is missing.
|
|
8
|
+
|
|
9
|
+
## What it is
|
|
10
|
+
|
|
11
|
+
- A command-line tool (`evalsutra`) that needs only Node.js 22 or newer. No API key, no account, no network, no dependencies.
|
|
12
|
+
- A set of 18 portable specialist **skills** (instruction files) that Claude Code, Cursor, Codex, Gemini CLI and GitHub Copilot can load, so the AI in your editor follows the same evidence rules.
|
|
13
|
+
- Reports in JSON, Markdown, HTML and JUnit, plus an independent-style **self-audit** report that shows the tool's own accuracy.
|
|
14
|
+
|
|
15
|
+
## What it is not
|
|
16
|
+
|
|
17
|
+
- It does **not** run your agent. It evaluates observations you supply. Live execution is on the roadmap, not in this release.
|
|
18
|
+
- It does **not** certify safety. `GO_CANDIDATE` means "recorded evidence met your thresholds", never "approved for production".
|
|
19
|
+
- It does **not** call any AI model, so it cannot be fooled by a persuasive answer.
|
|
20
|
+
|
|
21
|
+
## Why trust it
|
|
22
|
+
|
|
23
|
+
| Trust signal | Where to see it |
|
|
24
|
+
|---|---|
|
|
25
|
+
| Public source, Apache-2.0 | [GitHub repository](https://github.com/shahvatsalofficial/evalsutra) |
|
|
26
|
+
| Zero runtime dependencies | `package.json`, audit check S5 |
|
|
27
|
+
| No network or process APIs in source | audit check S4 |
|
|
28
|
+
| Test suite run on 3 operating systems × 3 Node versions | [GitHub Actions](https://github.com/shahvatsalofficial/evalsutra/actions) |
|
|
29
|
+
| 29-check self-audit with expected vs observed values | [Testing and evidence](08-testing-and-evidence.md), [Audit](09-audit-and-trust.md) |
|
|
30
|
+
| Raw test output and sample reports committed | `documents/evalsutra/evidence/` |
|
|
31
|
+
| Honest limits written down | [Roadmap and status](13-roadmap-and-status.md) |
|
|
32
|
+
|
|
33
|
+
## Business value
|
|
34
|
+
|
|
35
|
+
1. **Catch regressions before users do.** Compare two runs on the same cases; a pass-to-fail change fails the build.
|
|
36
|
+
2. **Stop score inflation.** Missing data is "inconclusive", never a pass. Model-judge opinions cannot prove a pass.
|
|
37
|
+
3. **Cheap to adopt.** One command, no vendor, no cost. Works inside CI and offline.
|
|
38
|
+
4. **Audit trail.** Every report carries SHA-256 digests of its inputs, tool version, Node version and platform.
|
|
39
|
+
|
|
40
|
+
## Current maturity
|
|
41
|
+
|
|
42
|
+
Version 0.4.0 is an early release (0.x). The evaluator, reports, installer, trace analysis, repro capsule and self-audit are implemented and tested. Live agent execution, sandboxing, a server, a dashboard and third-party integrations are **planned**. See [Roadmap and status](13-roadmap-and-status.md).
|
|
43
|
+
|
|
44
|
+
## Who should read what
|
|
45
|
+
|
|
46
|
+
| Reader | Start here |
|
|
47
|
+
|---|---|
|
|
48
|
+
| Executive, buyer | this page, then [Security and legal](07-security-and-legal.md) |
|
|
49
|
+
| Product, QA | [Functional specification](02-functional-specification.md), [User guide](05-user-guide.md) |
|
|
50
|
+
| Engineer, architect | [Architecture](03-architecture.md), [Technical reference](04-technical-reference.md) |
|
|
51
|
+
| Security, compliance | [Security and legal](07-security-and-legal.md), [Audit](09-audit-and-trust.md) |
|
|
52
|
+
| Contributor | [Contributor handbook](12-contributor-handbook.md) |
|