evalsutra 0.0.0-stage → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (121) hide show
  1. package/.agents/skills/evalsutra-arena/SKILL.md +27 -0
  2. package/.agents/skills/evalsutra-audit/SKILL.md +27 -0
  3. package/.agents/skills/evalsutra-bench/SKILL.md +27 -0
  4. package/.agents/skills/evalsutra-cost/SKILL.md +27 -0
  5. package/.agents/skills/evalsutra-data/SKILL.md +27 -0
  6. package/.agents/skills/evalsutra-drift/SKILL.md +27 -0
  7. package/.agents/skills/evalsutra-explain/SKILL.md +27 -0
  8. package/.agents/skills/evalsutra-guard/SKILL.md +27 -0
  9. package/.agents/skills/evalsutra-judge/SKILL.md +27 -0
  10. package/.agents/skills/evalsutra-pluginwatch/SKILL.md +27 -0
  11. package/.agents/skills/evalsutra-port/SKILL.md +27 -0
  12. package/.agents/skills/evalsutra-prime/SKILL.md +27 -0
  13. package/.agents/skills/evalsutra-report/SKILL.md +27 -0
  14. package/.agents/skills/evalsutra-robust/SKILL.md +27 -0
  15. package/.agents/skills/evalsutra-task/SKILL.md +27 -0
  16. package/.agents/skills/evalsutra-trace/SKILL.md +27 -0
  17. package/.agents/skills/evalsutra-truth/SKILL.md +27 -0
  18. package/.agents/skills/evalsutra-verify/SKILL.md +27 -0
  19. package/.claude-plugin/plugin.json +8 -0
  20. package/CHANGELOG.md +33 -0
  21. package/CITATION.cff +14 -0
  22. package/CODE_OF_CONDUCT.md +2 -0
  23. package/CONTRIBUTING.md +2 -0
  24. package/LICENSE +202 -0
  25. package/NOTICE +5 -0
  26. package/README.md +161 -2
  27. package/SECURITY.md +17 -0
  28. package/SUPPORT.md +5 -0
  29. package/THIRD_PARTY_NOTICES.md +14 -0
  30. package/docs/ADVANCED_TOOLING_ROADMAP.md +244 -0
  31. package/docs/ARCHITECTURE.md +17 -0
  32. package/docs/PLUGIN_AND_SKILL_CATALOG.md +56 -0
  33. package/docs/STATUS.md +25 -0
  34. package/docs/master-claude-prompt.md +534 -0
  35. package/documents/evalsutra/01-executive-summary.md +52 -0
  36. package/documents/evalsutra/02-functional-specification.md +104 -0
  37. package/documents/evalsutra/03-architecture.md +78 -0
  38. package/documents/evalsutra/04-technical-reference.md +150 -0
  39. package/documents/evalsutra/05-user-guide.md +94 -0
  40. package/documents/evalsutra/06-skills-catalog.md +52 -0
  41. package/documents/evalsutra/07-security-and-legal.md +66 -0
  42. package/documents/evalsutra/08-testing-and-evidence.md +63 -0
  43. package/documents/evalsutra/09-audit-and-trust.md +54 -0
  44. package/documents/evalsutra/10-ci-cd-and-release.md +50 -0
  45. package/documents/evalsutra/11-faq-and-glossary.md +51 -0
  46. package/documents/evalsutra/12-contributor-handbook.md +67 -0
  47. package/documents/evalsutra/13-roadmap-and-status.md +48 -0
  48. package/documents/evalsutra/assets/architecture.svg +21 -0
  49. package/documents/evalsutra/evidence/README.md +39 -0
  50. package/documents/evalsutra/evidence/samples/audit/audit-report.html +46 -0
  51. package/documents/evalsutra/evidence/samples/audit/audit-report.json +317 -0
  52. package/documents/evalsutra/evidence/samples/audit/audit-report.md +64 -0
  53. package/documents/evalsutra/evidence/samples/bad/junit.xml +2 -0
  54. package/documents/evalsutra/evidence/samples/bad/report.html +47 -0
  55. package/documents/evalsutra/evidence/samples/bad/report.json +153 -0
  56. package/documents/evalsutra/evidence/samples/bad/report.md +22 -0
  57. package/documents/evalsutra/evidence/samples/compare-good-vs-bad.json +59 -0
  58. package/documents/evalsutra/evidence/samples/demo-suite.json +13 -0
  59. package/documents/evalsutra/evidence/samples/gate-bad-run.json +10 -0
  60. package/documents/evalsutra/evidence/samples/good/junit.xml +2 -0
  61. package/documents/evalsutra/evidence/samples/good/report.html +47 -0
  62. package/documents/evalsutra/evidence/samples/good/report.json +153 -0
  63. package/documents/evalsutra/evidence/samples/good/report.md +22 -0
  64. package/documents/evalsutra/evidence/samples/trace-analysis.json +33 -0
  65. package/documents/evalsutra/evidence/test-run.txt +71 -0
  66. package/documents/evalsutra/evidence/test-summary.json +11 -0
  67. package/documents/evalsutra/html/01-executive-summary.html +57 -0
  68. package/documents/evalsutra/html/02-functional-specification.html +75 -0
  69. package/documents/evalsutra/html/03-architecture.html +60 -0
  70. package/documents/evalsutra/html/04-technical-reference.html +109 -0
  71. package/documents/evalsutra/html/05-user-guide.html +72 -0
  72. package/documents/evalsutra/html/06-skills-catalog.html +54 -0
  73. package/documents/evalsutra/html/07-security-and-legal.html +63 -0
  74. package/documents/evalsutra/html/08-testing-and-evidence.html +61 -0
  75. package/documents/evalsutra/html/09-audit-and-trust.html +61 -0
  76. package/documents/evalsutra/html/10-ci-cd-and-release.html +61 -0
  77. package/documents/evalsutra/html/11-faq-and-glossary.html +57 -0
  78. package/documents/evalsutra/html/12-contributor-handbook.html +71 -0
  79. package/documents/evalsutra/html/13-roadmap-and-status.html +51 -0
  80. package/documents/evalsutra/html/assets/architecture.svg +21 -0
  81. package/documents/evalsutra/html/index.html +42 -0
  82. package/examples/demo-observations-bad.json +7 -0
  83. package/examples/demo-observations.json +7 -0
  84. package/examples/observations.json +10 -0
  85. package/examples/output/report.json +99 -0
  86. package/examples/output/report.md +18 -0
  87. package/examples/suite.json +11 -0
  88. package/examples/traces/bad-trace.json +5 -0
  89. package/llms.txt +25 -0
  90. package/package.json +19 -4
  91. package/packages/audit/audit.mjs +179 -0
  92. package/packages/cli/evalsutra.mjs +151 -0
  93. package/packages/cli/install.mjs +68 -0
  94. package/packages/core/capsule.mjs +31 -0
  95. package/packages/core/evaluate.mjs +187 -0
  96. package/packages/core/security.mjs +22 -0
  97. package/packages/core/stats.mjs +16 -0
  98. package/packages/core/trace.mjs +29 -0
  99. package/packages/reporters/index.mjs +26 -0
  100. package/packages/reporters/theme.mjs +53 -0
  101. package/policies/README.md +3 -0
  102. package/policies/default.json +1 -0
  103. package/skills/evalsutra-arena/SKILL.md +27 -0
  104. package/skills/evalsutra-audit/SKILL.md +27 -0
  105. package/skills/evalsutra-bench/SKILL.md +27 -0
  106. package/skills/evalsutra-cost/SKILL.md +27 -0
  107. package/skills/evalsutra-data/SKILL.md +27 -0
  108. package/skills/evalsutra-drift/SKILL.md +27 -0
  109. package/skills/evalsutra-explain/SKILL.md +27 -0
  110. package/skills/evalsutra-guard/SKILL.md +27 -0
  111. package/skills/evalsutra-judge/SKILL.md +27 -0
  112. package/skills/evalsutra-pluginwatch/SKILL.md +27 -0
  113. package/skills/evalsutra-port/SKILL.md +27 -0
  114. package/skills/evalsutra-prime/SKILL.md +27 -0
  115. package/skills/evalsutra-report/SKILL.md +27 -0
  116. package/skills/evalsutra-robust/SKILL.md +27 -0
  117. package/skills/evalsutra-task/SKILL.md +27 -0
  118. package/skills/evalsutra-trace/SKILL.md +27 -0
  119. package/skills/evalsutra-truth/SKILL.md +27 -0
  120. package/skills/evalsutra-verify/SKILL.md +27 -0
  121. package/suites/demo.json +13 -0
@@ -0,0 +1,244 @@
1
+ # EvalSutra AI — Advanced Tooling, Differentiation & Integration Roadmap
2
+
3
+ **Project:** EvalSutra AI — Agent Evaluation & Reliability Intelligence Platform
4
+ **Creator:** Vatsal Shah
5
+ **Proposed namespace:** `evalsutra`
6
+ **Status:** Engineering plan. External connectors and advanced modules below are NOT implemented in the starter.
7
+ **Updated:** 2026-11-01
8
+
9
+ ## 0. Guiding product contract
10
+
11
+ The most valuable objective is an **agent reliability laboratory** rather than a wrapper around commercial evaluators. EvalSutra must measure demonstrable changes to an environment, tool correctness, repeatability, security, privacy, cost and cross-host behavior, using a **reproducible evidence record**.
12
+
13
+ **Non-negotiable:** The default CLI must work on Windows/macOS/Linux without model API keys, cloud account, remote service, container engine, Python runtime, or paid subscription. Offline JSON fixture evaluation, deterministic assertions, score comparisons and reports must keep working. Real agent execution, browser testing, cloud APIs, model judges and sandbox orchestration must be optional and capability-detected. An IDE subscription/session cannot be scraped or reused as a hidden API; use only documented host integrations.
14
+
15
+ **Disclosure:** A plugin, an SDK, a framework adapter, an optional locally hosted service and a CLI test runner are distinct things. Do not call every integration a 'Claude plugin'. List and verify actual compatibility, credentials, license, cost and permissions before installing anything.
16
+
17
+ ## 1. Additional tools beyond the previous catalog (priority matrix)
18
+
19
+ | Integration | Priority | What it contributes | Default-mode dependency? | Notes |
20
+ |---|---|---|---|---|
21
+ | **Harbor + Terminal-Bench** | P1 | Realistic containerized coding and terminal-agent benchmark harness | No | Official https://github.com/harbor-framework/harbor ; Docker/runtime and model authentication may be required for live runs. |
22
+ | **MCP Inspector** | P1 | Test MCP server tools, resources, prompts, schemas and error handling; CLI automation | No | Official https://github.com/modelcontextprotocol/inspector ; test only trusted / isolated MCP servers. |
23
+ | **OpenTelemetry GenAI conventions / OTLP** | P1 | Portable agent traces, spans, events, tokens and tool calls | No | https://opentelemetry.io/docs/specs/semconv/ ; GenAI conventions evolve; version the mapping. |
24
+ | **Opik** | P2 (choose one observability bridge first) | Self-hostable / remote trace & eval experiments, optional MCP integration | No | https://www.comet.com/docs/opik/ ; assess deployment, licensing and data handling. |
25
+ | **Arize Phoenix** | P2 alternative to Opik | Local agent trace inspection, datasets, experiments | No | https://arize.com/phoenix/ ; source-available ELv2, **not Apache-2.0**; do not bundle code without review. |
26
+ | **Microsoft PyRIT** | P2 | Authorized red-team and prompt-injection robustness scenarios | No | https://microsoft.github.io/PyRIT/ ; safe, controlled targets only. |
27
+ | **Hypothesis (Python) / fast-check (TypeScript)** | P1 | Property-based test generation of graders, parsers and adapters | Dev-only | https://hypothesis.readthedocs.io/ and https://fast-check.dev/ . |
28
+ | **uv** | P2 | Efficient optional Python tool and benchmark environment management | No | https://docs.astral.sh/uv/ ; never force Python on JS-only users. |
29
+ | **DuckDB / SQLite** | P1 optional storage backend | Local analytical runs/datasets, efficient aggregate comparisons | No | Use JSON flat-file fallback; SQLite for metadata, DuckDB for analytics only when justified. |
30
+ | **Gitleaks** | P1 CI | Detect accidentally committed secrets | No | https://github.com/gitleaks/gitleaks . |
31
+ | **OSV-Scanner** | P1 CI | Dependency vulnerability checking with actionable evidence | No | https://github.com/google/osv-scanner . |
32
+ | **OpenSSF Scorecard** | P2 CI | Public-repository security posture checks | No | https://github.com/ossf/scorecard . |
33
+ | **npm trusted publishing + provenance** | P2 publish | Safer official package publication via OIDC | No | https://docs.npmjs.com/trusted-publishers/ . |
34
+ | **Changesets or Release Please** | P2 publish, choose one | Versioning, changelogs, automated release PRs | No | Avoid adopting both. |
35
+ | **DVC** | P3 optional large-dataset workflows | Version large benchmark datasets without committing sensitive blobs | No | https://dvc.org/ ; prefer simple content hashes and Git for small suites. |
36
+ | **Ragas** | P3 specialized RAG evaluation | Retrieval-augmented agent grounding tests | No | https://docs.ragas.io/ ; activate only for RAG suites. |
37
+
38
+ **Already recommended; do not count as new or install twice:** Promptfoo, Inspect AI, Docker/Podman, Playwright, Ollama, GitHub Actions, Claude Code Setup, Superpowers, Code Review, Feature Dev, Security Guidance, GitHub, Context7, Serena, TypeScript LSP, PR Review Toolkit, Langfuse, LangSmith, MLflow and DeepEval. Treat the latter hosted or heavier frameworks as possible adapters, not required dependencies.
39
+
40
+ ### Tool decisions (avoid overengineering)
41
+
42
+ 1. Make **Harbor** the first live *coding-agent benchmark* adapter; keep existing deterministic replay as the default.
43
+ 2. Make **MCP Inspector** the first *protocol/tool testing* integration; define stable imported-result JSON and a CLI wrapper if supported.
44
+ 3. Implement **OpenTelemetry import/export** as the shared telemetry interchange; only **one** optional observability UI initially (choose Opik or Phoenix after proof-of-concept).
45
+ 4. Add **PyRIT** after access controls and isolated fixtures; use it for authorized safety tests, not unsupervised internet targets.
46
+ 5. Add **fast-check** (and Hypothesis only for the Python adapter) for parser and score-engine fuzz testing.
47
+ 6. Add **Gitleaks, OSV-Scanner, CodeQL** (where license/config allow), pinned CI actions and package provenance to public-release hardening.
48
+ 7. Do **not** force a heavy web dashboard or full cloud stack into the first milestone. CLI and HTML reports first; dashboard later.
49
+
50
+ ## 2. Distinctive product innovations to build
51
+
52
+ These are proposed proprietary-to-EvalSutra *features* in the sense of product differentiation, not claims of patentability or novel scientific algorithms.
53
+
54
+ **A. EvalSutra Repro Capsule (`repro`)**
55
+ A portable directory or archive containing suite definition and schema version; agent and adapter metadata; sanitized input and output; tool trace; redacted environment/OS/runtime fingerprints; execution timestamps; sanitized artifacts; hashes; and deterministic scoring configuration. A consumer can replay *grading* without rerunning an AI model. No secrets or credentials should be included. Use a manifest and SHA-256 digest per item; do not claim tamper-proofing unless signed and verified.
56
+
57
+ **B. EvalSutra Compat Doctor (`compat`)**
58
+ Host capability probes for Claude Code, Cursor, Codex, Gemini CLI, Copilot and generic CLI/MCP. Report only **verified**, **partially verified**, **unsupported**, or **unknown** per capability: skill discovery, agent invocation, tools, filesystem edits, traces, end-state verification, background execution, cost data and network policy. Do not say 'all IDEs supported' without tested contracts. Host-specific official paths must be version-checked.
59
+
60
+ **C. EvalSutra Trace Lens (`trace`)**
61
+ Trace parser that highlights unauthorized actions, skipped checks, tool-order errors, hallucinated success claims, redundant tool loops, timeouts, fallback behavior and tool-call failures. Provide direct evidence IDs and machine-readable discrepancy codes.
62
+
63
+ **D. EvalSutra Judge Calibration Lab (`calibrate`)**
64
+ Optional AI judges are tested against human-reviewed gold labels. Track label agreement, per-category disagreement, prompt/model versions, rubrics, class balance and judge cost. AI judge output is an opinion, **not** objective ground truth. No-key deterministic scoring must remain independent.
65
+
66
+ **E. EvalSutra Agent Scorecard (`scorecard`)**
67
+ A versioned scorecard that separates objective task completion, tool compliance, safety boundaries, reproducibility, latency and cost. Show missing data rather than treating missing observations as zero or success. Publish only accompanied by sample size, task categories, environment and confidence intervals when applicable.
68
+
69
+ **F. EvalSutra Regression Firewall (`gate`)**
70
+ Compare candidate and baseline on matched, version-pinned tasks. Severe confirmed policy violations or benchmark invalidity block automatically. For other scores, use explicit thresholds with uncertainty; abstain when insufficient evidence. Produce PASS, FAIL, or INCONCLUSIVE—never fabricate certainty.
71
+
72
+ **G. EvalSutra Benchmark Integrity (`integrity`)**
73
+ Detect fixture duplication, leaked answers, test contamination risks, dataset licensing ambiguity, path leakage, benchmark drift and suspicious harness tampering. Reserve blind holdout datasets, with sealed expectations stored separately from the agent's input files.
74
+
75
+ **H. EvalSutra Failure-to-Fix Loop (`triage`)**
76
+ Cluster failures by mechanism and produce evidence-backed issue drafts, reproduction commands and proposed fixes. Never rewrite agent code or submit PRs without explicit authorization. Verify any proposed change using a fresh, isolated rerun.
77
+
78
+ ## 3. Multi-level platform architecture
79
+
80
+ **Level 1 — Product/enterprise:** local CLI; independent packages; opt-in hosted enterprise add-on; open schema and exporter; cross-project compatibility standards.
81
+
82
+ **Level 2 — Solution:** `CLI / IDE Skill / Server API` -> `Capability Discovery` -> `Suite Registry` -> `Execution Adapter` -> `Isolated Runner` -> `Evidence Collector` -> `Deterministic Evaluators` -> `Optional AI Judge` -> `Comparison/Gating` -> `Reports/Artifacts`. Runs must have stable IDs and redacted logs. A failed or interrupted runner must preserve clear provenance.
83
+
84
+ **Level 3 — Component/module contracts:**
85
+
86
+ - `core/schema`: versioned JSON Schema validated at ingress; suite, fixture, observation, trace, result, policy and manifest.
87
+ - `core/engine`: pure deterministic evaluators with typed results and explicit inconclusive reasons.
88
+ - `core/statistics`: paired comparisons, confidence intervals, nondeterminism and stratified reports.
89
+ - `core/security`: redaction, permissions, secrets hygiene, provenance, size/time/network budgets.
90
+ - `runner`: read-only replay by default; opt-in sandbox execution with CPU/memory/time quotas, temporary workspace and outbound network denied by default.
91
+ - `adapters/host`: capability probes + execution contracts per IDE, never private token scraping.
92
+ - `integrations/harbor`, `/mcp-inspector`, `/otel`, `/opik`, `/phoenix`, `/pyrit` feature-flagged.
93
+ - `storage`: local JSON append-only artifacts; opt-in SQLite/DuckDB; no server required.
94
+ - `reports`: JSON, Markdown, HTML, JUnit, SARIF when meaningful, and optional OTLP export.
95
+ - `server`: optional API behind authenticated HTTPS, multi-tenant isolation, RBAC, scoped tokens, quotas, audit logs and queues.
96
+ - `dashboard`: optional accessible browser UI for task coverage, traces, comparisons, and release gates.
97
+
98
+ **Level 4 — Code-level:** Testable pure functions; time-independent deterministic score formulas; no `eval` or arbitrary commands from dataset definitions; validated input; size limits; abortable workers; deterministic fixture sorting and stable hash schemas.
99
+
100
+ ## 4. New file/folder layout to add (preserve the working starter)
101
+
102
+ ```text
103
+ evalsutra/
104
+ docs/
105
+ ADVANCED_TOOLING_ROADMAP.md
106
+ EVALUATION_METHODOLOGY.md # implement when relevant
107
+ BENCHMARK_INTEGRITY.md # implement when relevant
108
+ packages/
109
+ core/metrics/ # new tested metrics
110
+ core/manifest/ # repro capsule schema
111
+ core/trace/ # trace rules
112
+ core/compat/ # capability matrix
113
+ core/judge/ # optional and calibrated
114
+ runner/sandbox/ # optional verified isolation
115
+ statistics/ # comparative evaluation
116
+ reporters/ # append to existing reporter
117
+ server/ # optional: no placeholder API claims
118
+ dashboard/ # optional: later
119
+ integrations/
120
+ harbor/
121
+ mcp-inspector/
122
+ opentelemetry/
123
+ opik/ # initial observability choice
124
+ phoenix/ # alternative; keep off by default
125
+ pyrit/
126
+ suites/
127
+ fixtures/
128
+ gold/
129
+ adversarial/
130
+ compatibility/
131
+ schemas/
132
+ repro-manifest.schema.json
133
+ capability-report.schema.json
134
+ trace-event.schema.json
135
+ tests/
136
+ contract/
137
+ property/
138
+ security/
139
+ regression/
140
+ .github/
141
+ workflows/
142
+ security.yml
143
+ publish.yml # enable only with trusted publisher
144
+ ```
145
+
146
+ Create folders only when implementing actual files and testable features. Empty scaffolds must not be described as implementation. Keep `npm test`, `npm run doctor` and `npm run demo` working during migration.
147
+
148
+ ## 5. Phased build roadmap with objective gates
149
+
150
+ | Phase | Primary deliverable | Exit/acceptance evidence |
151
+ |---|---|---|
152
+ | **P0: Baseline** | Inventory starter, CI, threat model and full test pass | Existing tests green; present feature-status matrix; no regressions. |
153
+ | **P1: Portable evidence** | Repro Capsule, schema validation, trace-event normalization and redaction | Offline golden fixture survives export/import/replay; secrets never appear in output. |
154
+ | **P2: Trustworthy grading** | Repeatability, missing-data handling, paired comparisons and gate policies | Tests cover PASS/FAIL/INCONCLUSIVE; no AI key required; reproducible scores. |
155
+ | **P3: Protocol compatibility** | MCP Inspector adapter and host Compat Doctor | Feature matrix based on real smoke tests; unsupported capabilities reported honestly. |
156
+ | **P4: Real-world evaluation** | Harbor adapter + isolated runner | Known sample benchmark executed in approved sandbox; outcomes and sandbox telemetry captured; missing runtime returns actionable 'unsupported'. |
157
+ | **P5: Trace observability** | OpenTelemetry normalization, optional Opik/Phoenix adapter | Import + export round trip preserves event provenance; redaction verified. |
158
+ | **P6: Controlled AI judges** | Local Ollama or host-supported judge integration, calibration dataset | Human-reviewed gold comparison and judge uncertainty reporting; no key required for deterministic mode. |
159
+ | **P7: Safety & integrity** | PyRIT bridge, contamination checks, permission tests | Authorized sandbox-only adversarial tasks; no unauthorized network operations. |
160
+ | **P8: CI, server & dashboard** | Public workflows, optional secure API, accessible UI, release pipeline | OS matrix checks; threat-model and auth tests; artifact provenance; dashboard works without disclosing sensitive data. |
161
+ | **P9: Community release** | Contributing guide, plugin contracts, sample datasets, versioned documentation | Clean-clone install, examples, transparent support matrix, release assets and SBOM. |
162
+
163
+ **Recommended first public milestone:** Phases P0–P3 plus one small Harbor proof of concept. Defer costly live benchmarks and multi-tenant hosting until maintainers can sustain them.
164
+
165
+ ## 6. Benchmark and report design
166
+
167
+ ### Evaluation case structure
168
+
169
+ - `case_id`, `suite_version`, `input_ref`, `target_agent`, `host`, `platform`, `expected_end_state` (sealed as needed), `allowed_tools`, `forbidden_tools`, `timeout_budget`, `network_policy`, `grader_versions`, `policy_ref`, `baseline_ref`.
170
+ - Every observation declares `source_type`: `actual_run`, `imported_trace`, `human_label`, `test_fixture`, `model_judge`, or `unverified_claim`.
171
+ - Every result declares `status`: `pass`, `fail`, `inconclusive`, `skipped`, or `error`, plus `evidence_refs`, `reason_codes`, `confidence/uncertainty` as applicable.
172
+ - Score aggregation must **not** silently count missing, skipped, unrun or inconclusive cases as passes.
173
+
174
+ ### Multi-axis scorecard
175
+
176
+ Publish separate numbers for task success, tool correctness, authorization, robustness, latency, cost and repeatability. If presenting a composite score, show explicit weights and missing-data policy; default to no composite. Compare matched tasks, host versions, tool permissions and model modes; do not market rankings across incomparable setups.
177
+
178
+ ### Redaction and user consent
179
+
180
+ Use synthetic data by default. Opt-in before uploading prompts, traces, screenshots, proprietary documents or code. Mask credentials and personal data *before* third-party export. Define retention controls and deletion in server mode. Disallow network egress and host mounts for untrusted benchmark agents unless explicitly approved. Do not mistake Docker alone for a sufficient trust boundary for malicious workloads; document isolation assumptions.
181
+
182
+ ## 7. Tool review before installation
183
+
184
+ Claude must inventory actual installed plugins/MCP/CLI tools first. Evaluate publisher, maintenance, license, security advisories, version, data flow, telemetry, system requirements and cost. For each proposal state `required/optional`, `local/remote`, `free/paid potential`, `permission scope`, `installation command from verified docs`, `rollback command` and a minimal smoke test. Choose **project scope** by default. Never install globally or connect to external services without user permission. Do not run curl-to-shell installers blindly. Never store API keys in repository files. Use the user's IDE-native session only through published/authorized host capability.
185
+
186
+ ## 8. New CLI proposals (specification, not implemented)
187
+
188
+ ```sh
189
+ evalsutra repro export --run RUN_ID --out ./capsule
190
+ evalsutra repro verify ./capsule
191
+ evalsutra compat doctor --host claude-code
192
+ evalsutra trace analyze --input ./trace.json
193
+ evalsutra compare --baseline ./baseline --candidate ./candidate
194
+ evalsutra gate --policy ./policies/release.json
195
+ evalsutra benchmark run --adapter harbor --suite ./suites/coding --sandbox approved
196
+ evalsutra mcp inspect --config ./fixtures/mcp.json
197
+ evalsutra judge calibrate --gold ./suites/gold --provider local
198
+ evalsutra toolchain doctor
199
+ ```
200
+
201
+ Commands are proposed; Claude must build them before documenting as working. Failure codes must be stable and meaningful; distinguish unavailable runtime, failed test, invalid evidence, and genuine security violation.
202
+
203
+ ## 9. Suggested release quality gates
204
+
205
+ 1. Deterministic core, CLI and format tests pass in a **Windows/macOS/Linux** CI matrix; absence of CI evidence means partial verification.
206
+ 2. Replay succeeds without external credentials on a clean machine with the documented runtime installed.
207
+ 3. Security scanning finds no unresolved **confirmed critical** defects; severity and triage processes documented.
208
+ 4. Eval schemas and data migration paths are versioned and backward compatibility tested.
209
+ 5. A malicious fixture cannot run shell commands merely by being parsed; untrusted agent code only executes via an approved isolation layer.
210
+ 6. Judge evaluation never overrides a confirmed objective failure or prohibited action.
211
+ 7. Reports state exactly which capabilities and benchmarks were run and expose the reproducibility capsule.
212
+ 8. Starter examples include success, failure, inconclusive, timeout, missing evidence, tool mismatch and policy violation.
213
+ 9. Public benchmarks disclose suite versions, sample sizes, environment, runtime, provider/model mode, cost collection method and uncertainty.
214
+ 10. Licenses, SBOM, dependency checks, signed/provenance release options, security disclosure and contributor processes are documented.
215
+
216
+ ## 10. Special rules for Claude implementation
217
+
218
+ - Read the existing master prompt **plus this document**; both apply. If scopes conflict, preserve safety and working deterministic core; record an ADR for design choices.
219
+ - Never claim plugin installation, IDE support, sandbox enforcement, model evaluation, server deployment or test execution until verified in the running environment.
220
+ - Use existing EvalSutra agents: Prime owns orchestration; Bench owns suites; Trace owns tools; Port owns compatibility; Guard owns safety; Judge owns calibration; Drift owns regression; Verify owns independent acceptance; Report owns evidence display. Do not create overlapping new agents simply to add features.
221
+ - Before writing large new modules, implement a thin vertical slice with fixtures, schema, CLI, tests and example output.
222
+ - Create evidence-rich PR-ready commits or patches; reserve network access and global changes for explicit authorization.
223
+ - In the final engineering report list: `completed`, `tested`, `not tested`, `planned`, `blocked`, `requires consent`, and the precise test commands and results.
224
+
225
+ ## 11. Official references (re-verify before installation)
226
+
227
+ - Harbor: https://github.com/harbor-framework/harbor
228
+ - Terminal Bench: https://www.tbench.ai/
229
+ - Inspect AI eval registry: https://inspect.aisi.org.uk/evals/
230
+ - MCP Inspector: https://github.com/modelcontextprotocol/inspector
231
+ - OpenTelemetry: https://opentelemetry.io/docs/specs/semconv/
232
+ - Opik: https://www.comet.com/docs/opik/
233
+ - Phoenix: https://arize.com/phoenix/
234
+ - PyRIT: https://microsoft.github.io/PyRIT/
235
+ - Claude plugins: https://claude.com/marketplace/plugins
236
+ - npm trusted publishing: https://docs.npmjs.com/trusted-publishers/
237
+ - OpenSSF Scorecard: https://github.com/ossf/scorecard
238
+ - Gitleaks: https://github.com/gitleaks/gitleaks
239
+ - OSV-Scanner: https://github.com/google/osv-scanner
240
+ - GitHub Actions: https://docs.github.com/en/actions
241
+
242
+ ---
243
+
244
+ **Publication discipline:** The starter is not a finished multi-host agent-evaluation platform. Until tests exist, all additions remain proposed. Keep marketing honest, developer trust high, and every benchmark verifiable.
@@ -0,0 +1,17 @@
1
+ # Starter architecture
2
+
3
+ ```mermaid
4
+ flowchart LR
5
+ A[Suite JSON] --> C[Deterministic evaluator]
6
+ B[Recorded observations JSON] --> C
7
+ C --> D[Assertion-level evidence]
8
+ D --> E[Coverage and quality gate]
9
+ E --> F[JSON and Markdown reports]
10
+ F --> G[Baseline comparison]
11
+ H[Portable SKILL.md] --> I[Project-local IDE skills]
12
+ ```
13
+
14
+ The CLI uses only Node.js standard modules. It reads suites and recorded results from disk and does not call an LLM, execute an agent, contact a server, or modify the user's IDE configuration unless explicitly given `install`. The live-runner interface, sandbox, plugin adapters, server API, statistics and provenance hashing are targets for future implementation and documented in the master prompt.
15
+
16
+ ## Evidence limitations
17
+ Imported observations may be inaccurate or adversarial. The starter verifies assertions **against imported data** only; it cannot independently authenticate the underlying trace. For production, require attestation, read-only artifact stores and controlled runners.
@@ -0,0 +1,56 @@
1
+ # Plugins, Skills, MCP Servers and Developer Tool Catalog
2
+
3
+ This is a **recommendation list**, not a record of software installed on this machine. Check the actual Claude/IDE environment, policies, credentials and licenses first. Installation always needs consent for global tools, executables, MCP servers, accounts and cloud connections.
4
+
5
+ ## Claude Code official/marketplace plugins
6
+
7
+ | Name | Priority | Why |
8
+ |---|---|---|
9
+ | Superpowers | Essential | TDD, debugging, engineering process |
10
+ | Claude Code Setup | Essential | Recommend capabilities appropriate to this repository |
11
+ | Feature Dev | Essential | Exploration/implementation/review workflow |
12
+ | Code Review | Essential | Independent code reviews |
13
+ | Security Guidance | Essential | Defensive coding guidance |
14
+ | GitHub | Publish workflow | Issues, PRs and repository management |
15
+ | Context7 | Recommended | Current library documentation |
16
+ | Playwright | Recommended | Controlled browser end-state evaluation |
17
+ | TypeScript LSP | Recommended | Static navigation and type assistance |
18
+ | Serena | Recommended | Semantic codebase exploration |
19
+ | PR Review Toolkit | Recommended | Multi-dimensional PR assessment |
20
+ | CLAUDE.md Management | Recommended | Maintain agent instructions |
21
+ | Code Simplifier | Optional | Refactor readability |
22
+ | Commit Commands | Optional | Git workflow shortcuts |
23
+
24
+ Source: https://claude.com/marketplace/plugins and https://support.claude.com/en/articles/13837440-use-plugins-in-claude
25
+
26
+ ## Evaluation tooling, packages and optional services
27
+
28
+ | Tool | Role | Key needed? |
29
+ |---|---|---|
30
+ | Promptfoo | Assertions, traces, agent evals, optional red teaming | Depends on provider mode; local fixtures can be keyless |
31
+ | Promptfoo Eval Skills | CLI skills for agent eval authoring | No key to install skills; live model calls may need auth |
32
+ | Inspect AI | Agent benchmarks and sandbox tests | Depends on target model/runner |
33
+ | OpenTelemetry | Interoperable traces | No API key for local collection |
34
+ | Playwright | Browser state verification | No AI key; browser dependencies required |
35
+ | Docker/Podman | Process/container isolation | No AI key; installation and privileges required |
36
+ | Ollama | Local model adapter | No cloud key; local compute/model downloads |
37
+ | GitHub Actions | Cross-OS CI tests | GitHub repository/service account required |
38
+ | Langfuse, LangSmith, MLflow, DeepEval | Future optional telemetry/eval bridges | Depends on deployment/product |
39
+ | OpenAI, Anthropic, Gemini cloud APIs | Future optional server runners or judges | Approved credentials/billing generally required |
40
+
41
+ Sources: https://www.promptfoo.dev/docs/integrations/agent-skill/ ; https://inspect.aisi.org.uk/agents.html ; https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/ ; https://docs.github.com/en/actions
42
+
43
+ ## Required install safety workflow
44
+ 1. Run version and installed-plugin inventory commands using *documented* CLI interfaces only.
45
+ 2. Identify actual missing capability; avoid redundant plugins.
46
+ 3. Verify official publisher/repository, version, extension permissions, network dependencies, license and maintenance.
47
+ 4. Prefer project-scoped installation; show exact actions and obtain user approval for executable installations, global changes, external account connections and billing.
48
+ 5. Smoke-test installation; document removal and fallback instructions.
49
+ 6. Keep default replay mode fully functional with zero extra plugins or keys.
50
+
51
+ ## Cross-IDE distinction
52
+ Portable skills are not full execution engines. Claude: `.claude/skills`; Cursor: `.cursor/skills` or `.agents/skills`; Codex: `.agents/skills`; Gemini: `.gemini/skills` or `.agents/skills`; Copilot: `.github/skills` or `.agents/skills`. Support and host versions must be verified individually, and platform permissions respected.
53
+
54
+
55
+ ## November 2026 advanced tooling extension
56
+ The current authoritative additional tools, keyless/paid distinctions, risks, and rollout phases are in [`ADVANCED_TOOLING_ROADMAP.md`](ADVANCED_TOOLING_ROADMAP.md). New core priorities: Harbor, MCP Inspector, OpenTelemetry, property-based testing and Gitleaks/OSV; optional later: Opik *or* Phoenix, PyRIT, uv, local analytical stores, npm trusted publication. Never install every plugin without a use case.
package/docs/STATUS.md ADDED
@@ -0,0 +1,25 @@
1
+ # Feature status (v0.4.0)
2
+ Superseded in detail by documents/evalsutra/13-roadmap-and-status.md.
3
+
4
+ Verified locally on Windows 11, Node 24 (`npm test`: 44 pass, 1 skipped symlink test that is POSIX-only). Linux/macOS results come only from real CI runs.
5
+
6
+ | Capability | Status | Evidence |
7
+ |---|---|---|
8
+ | Offline replay, schema checks | Implemented | tests, `npm run demo` |
9
+ | Assertions: equals, normalized_equals, contains, not_contains, regex (length/nesting guard), json_has_keys | Implemented | unit tests |
10
+ | Tool required/forbidden/order, max_steps, max_latency_ms | Implemented | unit tests |
11
+ | Statuses pass/fail/inconclusive/error/skipped/not_supported + reason codes | Implemented | unit tests |
12
+ | Evidence source types; model_judge/unverified_claim can never prove a pass | Implemented | unit test |
13
+ | Gate NO_GO / REVIEW / GO_CANDIDATE, `gate --policy` | Implemented | CLI tests |
14
+ | Wilson CI, exact sign test on paired comparison, pass@k helper | Implemented | unit tests |
15
+ | Compare verdict PASS / FAIL / INCONCLUSIVE with confounders | Implemented | unit tests |
16
+ | Trace analysis (forbidden/unauthorized tool, loops, unsupported success claim, secrets) | Implemented (heuristic) | unit tests |
17
+ | Repro capsule export/verify (SHA-256, redacted, unsigned) | Implemented | CLI test |
18
+ | Reports JSON, Markdown, HTML (escaped), JUnit | Implemented | tests |
19
+ | Installer for 18 skills: dry-run, no-clobber, uninstall with change detection, symlink guard | Implemented | CLI tests |
20
+ | 18 agent skills | Instructions only; host AI executes; CLI support listed per skill | SKILL.md |
21
+ | Host skill discovery in live IDEs | Unverified | `evalsutra matrix` says unknown |
22
+ | Live agent runner, sandbox, server, dashboard, SARIF, OTEL, Promptfoo/Inspect/Harbor/MCP Inspector, judge calibration, audit-plugin, JSON Schema files | Not implemented | planned |
23
+ | Windows CI / macOS CI | Configured; see GitHub Actions for real results | |
24
+
25
+ `GO_CANDIDATE` is a recommendation on recorded evidence, not deployment approval. No target agent is run by EvalSutra.