evalsutra 0.0.0-stage → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/evalsutra-arena/SKILL.md +27 -0
- package/.agents/skills/evalsutra-audit/SKILL.md +27 -0
- package/.agents/skills/evalsutra-bench/SKILL.md +27 -0
- package/.agents/skills/evalsutra-cost/SKILL.md +27 -0
- package/.agents/skills/evalsutra-data/SKILL.md +27 -0
- package/.agents/skills/evalsutra-drift/SKILL.md +27 -0
- package/.agents/skills/evalsutra-explain/SKILL.md +27 -0
- package/.agents/skills/evalsutra-guard/SKILL.md +27 -0
- package/.agents/skills/evalsutra-judge/SKILL.md +27 -0
- package/.agents/skills/evalsutra-pluginwatch/SKILL.md +27 -0
- package/.agents/skills/evalsutra-port/SKILL.md +27 -0
- package/.agents/skills/evalsutra-prime/SKILL.md +27 -0
- package/.agents/skills/evalsutra-report/SKILL.md +27 -0
- package/.agents/skills/evalsutra-robust/SKILL.md +27 -0
- package/.agents/skills/evalsutra-task/SKILL.md +27 -0
- package/.agents/skills/evalsutra-trace/SKILL.md +27 -0
- package/.agents/skills/evalsutra-truth/SKILL.md +27 -0
- package/.agents/skills/evalsutra-verify/SKILL.md +27 -0
- package/.claude-plugin/plugin.json +8 -0
- package/CHANGELOG.md +21 -0
- package/CITATION.cff +14 -0
- package/LICENSE +202 -0
- package/NOTICE +5 -0
- package/README.md +160 -2
- package/SECURITY.md +17 -0
- package/THIRD_PARTY_NOTICES.md +14 -0
- package/docs/ADVANCED_TOOLING_ROADMAP.md +244 -0
- package/docs/ARCHITECTURE.md +17 -0
- package/docs/PLUGIN_AND_SKILL_CATALOG.md +56 -0
- package/docs/STATUS.md +25 -0
- package/docs/master-claude-prompt.md +534 -0
- package/documents/evalsutra/01-executive-summary.md +52 -0
- package/documents/evalsutra/02-functional-specification.md +104 -0
- package/documents/evalsutra/03-architecture.md +78 -0
- package/documents/evalsutra/04-technical-reference.md +143 -0
- package/documents/evalsutra/05-user-guide.md +96 -0
- package/documents/evalsutra/06-skills-catalog.md +52 -0
- package/documents/evalsutra/07-security-and-legal.md +66 -0
- package/documents/evalsutra/08-testing-and-evidence.md +63 -0
- package/documents/evalsutra/09-audit-and-trust.md +54 -0
- package/documents/evalsutra/10-ci-cd-and-release.md +48 -0
- package/documents/evalsutra/11-faq-and-glossary.md +51 -0
- package/documents/evalsutra/12-contributor-handbook.md +67 -0
- package/documents/evalsutra/13-roadmap-and-status.md +48 -0
- package/documents/evalsutra/assets/architecture.svg +21 -0
- package/documents/evalsutra/evidence/README.md +39 -0
- package/documents/evalsutra/evidence/samples/audit/audit-report.html +46 -0
- package/documents/evalsutra/evidence/samples/audit/audit-report.json +317 -0
- package/documents/evalsutra/evidence/samples/audit/audit-report.md +64 -0
- package/documents/evalsutra/evidence/samples/bad/junit.xml +2 -0
- package/documents/evalsutra/evidence/samples/bad/report.html +47 -0
- package/documents/evalsutra/evidence/samples/bad/report.json +153 -0
- package/documents/evalsutra/evidence/samples/bad/report.md +22 -0
- package/documents/evalsutra/evidence/samples/compare-good-vs-bad.json +59 -0
- package/documents/evalsutra/evidence/samples/demo-suite.json +13 -0
- package/documents/evalsutra/evidence/samples/gate-bad-run.json +10 -0
- package/documents/evalsutra/evidence/samples/good/junit.xml +2 -0
- package/documents/evalsutra/evidence/samples/good/report.html +47 -0
- package/documents/evalsutra/evidence/samples/good/report.json +153 -0
- package/documents/evalsutra/evidence/samples/good/report.md +22 -0
- package/documents/evalsutra/evidence/samples/trace-analysis.json +33 -0
- package/documents/evalsutra/evidence/test-run.txt +69 -0
- package/documents/evalsutra/evidence/test-summary.json +11 -0
- package/documents/evalsutra/html/01-executive-summary.html +57 -0
- package/documents/evalsutra/html/02-functional-specification.html +75 -0
- package/documents/evalsutra/html/03-architecture.html +60 -0
- package/documents/evalsutra/html/04-technical-reference.html +103 -0
- package/documents/evalsutra/html/05-user-guide.html +74 -0
- package/documents/evalsutra/html/06-skills-catalog.html +54 -0
- package/documents/evalsutra/html/07-security-and-legal.html +63 -0
- package/documents/evalsutra/html/08-testing-and-evidence.html +61 -0
- package/documents/evalsutra/html/09-audit-and-trust.html +61 -0
- package/documents/evalsutra/html/10-ci-cd-and-release.html +60 -0
- package/documents/evalsutra/html/11-faq-and-glossary.html +57 -0
- package/documents/evalsutra/html/12-contributor-handbook.html +71 -0
- package/documents/evalsutra/html/13-roadmap-and-status.html +51 -0
- package/documents/evalsutra/html/assets/architecture.svg +21 -0
- package/documents/evalsutra/html/index.html +42 -0
- package/examples/demo-observations-bad.json +7 -0
- package/examples/demo-observations.json +7 -0
- package/examples/observations.json +10 -0
- package/examples/output/report.json +99 -0
- package/examples/output/report.md +18 -0
- package/examples/suite.json +11 -0
- package/examples/traces/bad-trace.json +5 -0
- package/llms.txt +25 -0
- package/package.json +19 -4
- package/packages/audit/audit.mjs +179 -0
- package/packages/cli/evalsutra.mjs +145 -0
- package/packages/cli/install.mjs +68 -0
- package/packages/core/capsule.mjs +31 -0
- package/packages/core/evaluate.mjs +187 -0
- package/packages/core/security.mjs +22 -0
- package/packages/core/stats.mjs +16 -0
- package/packages/core/trace.mjs +29 -0
- package/packages/reporters/index.mjs +26 -0
- package/packages/reporters/theme.mjs +53 -0
- package/policies/README.md +3 -0
- package/policies/default.json +1 -0
- package/skills/evalsutra-arena/SKILL.md +27 -0
- package/skills/evalsutra-audit/SKILL.md +27 -0
- package/skills/evalsutra-bench/SKILL.md +27 -0
- package/skills/evalsutra-cost/SKILL.md +27 -0
- package/skills/evalsutra-data/SKILL.md +27 -0
- package/skills/evalsutra-drift/SKILL.md +27 -0
- package/skills/evalsutra-explain/SKILL.md +27 -0
- package/skills/evalsutra-guard/SKILL.md +27 -0
- package/skills/evalsutra-judge/SKILL.md +27 -0
- package/skills/evalsutra-pluginwatch/SKILL.md +27 -0
- package/skills/evalsutra-port/SKILL.md +27 -0
- package/skills/evalsutra-prime/SKILL.md +27 -0
- package/skills/evalsutra-report/SKILL.md +27 -0
- package/skills/evalsutra-robust/SKILL.md +27 -0
- package/skills/evalsutra-task/SKILL.md +27 -0
- package/skills/evalsutra-trace/SKILL.md +27 -0
- package/skills/evalsutra-truth/SKILL.md +27 -0
- package/skills/evalsutra-verify/SKILL.md +27 -0
- package/suites/demo.json +13 -0
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-arena
|
|
3
|
+
description: Compare agents on matched tasks with adequate samples and transparent trade-offs.
|
|
4
|
+
---
|
|
5
|
+
# Comparative Evaluation Analyst — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Compare agents on matched tasks with adequate samples and transparent trade-offs.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): compare. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-audit
|
|
3
|
+
description: Inspect grader logic, evidence authenticity, repeatability, score validity and risk of gaming.
|
|
4
|
+
---
|
|
5
|
+
# Evaluation Integrity Auditor — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Inspect grader logic, evidence authenticity, repeatability, score validity and risk of gaming.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): repro verify, validate. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-bench
|
|
3
|
+
description: Construct controlled suites, version fixtures and validate ground truths.
|
|
4
|
+
---
|
|
5
|
+
# Benchmark Engineer — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Construct controlled suites, version fixtures and validate ground truths.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): init, validate. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-cost
|
|
3
|
+
description: Analyze recorded token, latency, throughput and cost measurements with units and unknowns.
|
|
4
|
+
---
|
|
5
|
+
# Efficiency Evaluator — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Analyze recorded token, latency, throughput and cost measurements with units and unknowns.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): run (max_latency_ms only; token cost not implemented). Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-data
|
|
3
|
+
description: Verify fixture provenance, privacy, licensing and test leakage controls.
|
|
4
|
+
---
|
|
5
|
+
# Benchmark Data Steward — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Verify fixture provenance, privacy, licensing and test leakage controls.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): repro export/verify (hashes), validate. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-drift
|
|
3
|
+
description: Compare controlled baselines and candidates; identify regressions and confounding changes.
|
|
4
|
+
---
|
|
5
|
+
# Regression and Drift Analyst — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Compare controlled baselines and candidates; identify regressions and confounding changes.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): compare. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-explain
|
|
3
|
+
description: Correlate failed assertions with relevant evidence and recommend targeted investigation.
|
|
4
|
+
---
|
|
5
|
+
# Failure Diagnosis Specialist — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Correlate failed assertions with relevant evidence and recommend targeted investigation.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): report, trace analyze. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-guard
|
|
3
|
+
description: Check synthetic prompt injection, tool abuse and secret handling inside authorized isolated fixtures.
|
|
4
|
+
---
|
|
5
|
+
# Permission and Security Evaluator — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Check synthetic prompt injection, tool abuse and secret handling inside authorized isolated fixtures.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): trace analyze --forbid, run (not_contains canary). Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-judge
|
|
3
|
+
description: Define rubric criteria and optional LLM grader bias checks; never pass unsupported findings.
|
|
4
|
+
---
|
|
5
|
+
# Rubric and Judge Calibration Specialist — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Define rubric criteria and optional LLM grader bias checks; never pass unsupported findings.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): none: judge output is labeled model_judge and can never prove a pass. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-pluginwatch
|
|
3
|
+
description: Review plugin manifests and permissions statically; never install or execute untrusted code.
|
|
4
|
+
---
|
|
5
|
+
# Agent Plugin Safety Reviewer — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Review plugin manifests and permissions statically; never install or execute untrusted code.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): none: manual review checklist only; audit-plugin not implemented. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-port
|
|
3
|
+
description: Report verified capabilities and limitations across IDE, operating system and runner adapters.
|
|
4
|
+
---
|
|
5
|
+
# Compatibility Evaluator — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Report verified capabilities and limitations across IDE, operating system and runner adapters.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): matrix, install --dry-run. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-prime
|
|
3
|
+
description: Plan scoped evaluations, select objective measurements, coordinate specialist reviewers and deliver conservative evidence-based gates.
|
|
4
|
+
---
|
|
5
|
+
# Chief Evaluation Architect — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Plan scoped evaluations, select objective measurements, coordinate specialist reviewers and deliver conservative evidence-based gates.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): run, gate, report, matrix. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-report
|
|
3
|
+
description: Translate structured results into accessible actionable reports without changing facts.
|
|
4
|
+
---
|
|
5
|
+
# Evaluation Report Engineer — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Translate structured results into accessible actionable reports without changing facts.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): report --format md|html|junit. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-robust
|
|
3
|
+
description: Assess interruption, timeout, retry, escalation and failure recovery evidence.
|
|
4
|
+
---
|
|
5
|
+
# Resilience Evaluator — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Assess interruption, timeout, retry, escalation and failure recovery evidence.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): run (max_latency_ms/max_steps), trace analyze. Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-task
|
|
3
|
+
description: Check actual task artifacts and externally verified outcomes instead of polished responses.
|
|
4
|
+
---
|
|
5
|
+
# Task Outcome Verifier — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Check actual task artifacts and externally verified outcomes instead of polished responses.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): run (supply postcondition output as observations). Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-trace
|
|
3
|
+
description: Analyze recorded tool calls, ordering, spans, action permissions and missing trace fields.
|
|
4
|
+
---
|
|
5
|
+
# Trace Investigator — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Analyze recorded tool calls, ordering, spans, action permissions and missing trace fields.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): trace analyze, run (tool assertions). Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-truth
|
|
3
|
+
description: Validate answers against supplied trusted references; distinguish unsupported claims.
|
|
4
|
+
---
|
|
5
|
+
# Grounding and Factuality Reviewer — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Validate answers against supplied trusted references; distinguish unsupported claims.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): run (equals/normalized_equals/regex assertions). Everything else is manual. See `docs/STATUS.md`.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: evalsutra-verify
|
|
3
|
+
description: Re-run determinate validations; verify critical gates and isolate unsupported results.
|
|
4
|
+
---
|
|
5
|
+
# Independent Verification Specialist — EvalSutra AI
|
|
6
|
+
|
|
7
|
+
Use this skill when a developer requests evaluation of AI agents, skills, prompts, tools, or multi-agent workflows related to this specialty.
|
|
8
|
+
|
|
9
|
+
## Objective
|
|
10
|
+
Re-run determinate validations; verify critical gates and isolate unsupported results.
|
|
11
|
+
|
|
12
|
+
## Procedure
|
|
13
|
+
1. Confirm the target, intended task, permission boundary and available evidence.
|
|
14
|
+
2. Prefer objective postconditions and reproducible deterministic assertions; distinguish replay from actual execution.
|
|
15
|
+
3. Identify unsupported host/runtime capabilities and label missing data as inconclusive.
|
|
16
|
+
4. Use existing authorized evaluation runners; require explicit consent for live execution, network and external services.
|
|
17
|
+
5. Never edit benchmark ground truth, fabricate scores, access real secrets, or silently change security settings.
|
|
18
|
+
6. Record affected case ID, evidence source, method, version, status, confidence or uncertainty, and suggested safe remediation.
|
|
19
|
+
7. Return a concise findings summary with independent verification steps and limitations.
|
|
20
|
+
|
|
21
|
+
## Contract
|
|
22
|
+
Input: scope + suite/trace/artifacts + explicit permissions.
|
|
23
|
+
Output: structured evidence-backed findings and reproducibility notes.
|
|
24
|
+
Tools: host-permitted read-only analysis and explicitly authorized evaluation commands.
|
|
25
|
+
Stop conditions: absent authorization; destructive or credential-exposing command; unknown external data handling.
|
|
26
|
+
|
|
27
|
+
**Status:** Portable instructions executed by the host AI. EvalSutra CLI support (v0.2): run, gate, repro verify. Everything else is manual. See `docs/STATUS.md`.
|
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.3.0 (2026-11-07)
|
|
4
|
+
- Added `evalsutra audit`: 29-check self-audit with Markdown, HTML and JSON reports (grader accuracy on 36 known answers, evidence rules, reproducibility, security, skills, legal).
|
|
5
|
+
- Reports now use a shared accessible design system (light/dark, no script, status as text plus symbol); HTML report shows provenance and confidence interval.
|
|
6
|
+
- Added documentation set in documents/evalsutra/ (13 Markdown pages with generated styled HTML) and generated evidence (test run, sample reports, audit).
|
|
7
|
+
- Added NOTICE, THIRD_PARTY_NOTICES.md, CITATION.cff, SUPPORT.md, llms.txt, .gitattributes; rewrote README and SECURITY.md.
|
|
8
|
+
- Fixed: case ids such as "constructor" no longer read inherited object members; `init` no longer leaves a half-written folder; `trace analyze` requires --input; removed a hard-coded doctor field.
|
|
9
|
+
- CI: Node 24 added, audit and docs-freshness steps added, audit report uploaded as artifact.
|
|
10
|
+
- Fixed CI: `npm test` now uses `node --test` (glob failed on Windows + Node 20).
|
|
11
|
+
|
|
12
|
+
## 0.2.0
|
|
13
|
+
- Added assertions normalized_equals, regex, json_has_keys; statuses error/skipped/not_supported with reason codes.
|
|
14
|
+
- Added evidence source types; weak sources cannot prove a pass.
|
|
15
|
+
- Added Wilson CI, paired sign test, compare verdicts, `gate`, `report`, `trace analyze`, `repro export/verify`, `list-skills`, `matrix`, `init`, `uninstall`.
|
|
16
|
+
- Installer now installs all 18 skills with dry-run, lock file and symlink guard.
|
|
17
|
+
- HTML and JUnit reporters; digests and provenance in reports.
|
|
18
|
+
- Exit codes: 0 ok, 1 error, 2 NO_GO/FAIL, 3 REVIEW/INCONCLUSIVE, 4 invalid evidence.
|
|
19
|
+
|
|
20
|
+
## 0.1.0
|
|
21
|
+
- Starter: offline replay evaluator, Prime skill installer.
|
package/CITATION.cff
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
cff-version: 1.2.0
|
|
2
|
+
message: "If you use EvalSutra AI, please cite it as below."
|
|
3
|
+
title: "EvalSutra AI: The Agent Evaluation & Reliability Intelligence Platform"
|
|
4
|
+
type: software
|
|
5
|
+
version: 0.3.0
|
|
6
|
+
date-released: 2026-11-07
|
|
7
|
+
license: Apache-2.0
|
|
8
|
+
repository-code: "https://github.com/shahvatsalofficial/evalsutra"
|
|
9
|
+
url: "https://github.com/shahvatsalofficial/evalsutra"
|
|
10
|
+
abstract: "Evidence-first, offline, keyless evaluation of AI agents: deterministic graders, honest inconclusive handling, release gates, regression comparison, trace analysis, reproducibility capsules and a self-audit."
|
|
11
|
+
keywords: [ai-agents, agent-evaluation, llm-evals, agent-skills, claude-code, reliability, testing]
|
|
12
|
+
authors:
|
|
13
|
+
- family-names: Shah
|
|
14
|
+
given-names: Vatsal
|