@trohde/earos 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +156 -0
- package/assets/init/.agents/skills/earos-artifact-gen/SKILL.md +106 -0
- package/assets/init/.agents/skills/earos-artifact-gen/references/interview-guide.md +313 -0
- package/assets/init/.agents/skills/earos-artifact-gen/references/output-guide.md +367 -0
- package/assets/init/.agents/skills/earos-assess/SKILL.md +212 -0
- package/assets/init/.agents/skills/earos-assess/references/calibration-benchmarks.md +160 -0
- package/assets/init/.agents/skills/earos-assess/references/output-templates.md +311 -0
- package/assets/init/.agents/skills/earos-assess/references/scoring-protocol.md +281 -0
- package/assets/init/.agents/skills/earos-calibrate/SKILL.md +153 -0
- package/assets/init/.agents/skills/earos-calibrate/references/agreement-metrics.md +188 -0
- package/assets/init/.agents/skills/earos-calibrate/references/calibration-protocol.md +263 -0
- package/assets/init/.agents/skills/earos-create/SKILL.md +257 -0
- package/assets/init/.agents/skills/earos-create/references/criterion-writing-guide.md +268 -0
- package/assets/init/.agents/skills/earos-create/references/dependency-rules.md +193 -0
- package/assets/init/.agents/skills/earos-create/references/rubric-interview-guide.md +123 -0
- package/assets/init/.agents/skills/earos-create/references/validation-checklist.md +238 -0
- package/assets/init/.agents/skills/earos-profile-author/SKILL.md +251 -0
- package/assets/init/.agents/skills/earos-profile-author/references/criterion-writing-guide.md +280 -0
- package/assets/init/.agents/skills/earos-profile-author/references/design-methods.md +158 -0
- package/assets/init/.agents/skills/earos-profile-author/references/profile-checklist.md +173 -0
- package/assets/init/.agents/skills/earos-remediate/SKILL.md +118 -0
- package/assets/init/.agents/skills/earos-remediate/references/output-template.md +199 -0
- package/assets/init/.agents/skills/earos-remediate/references/remediation-patterns.md +330 -0
- package/assets/init/.agents/skills/earos-report/SKILL.md +85 -0
- package/assets/init/.agents/skills/earos-report/references/portfolio-template.md +181 -0
- package/assets/init/.agents/skills/earos-report/references/single-artifact-template.md +168 -0
- package/assets/init/.agents/skills/earos-review/SKILL.md +130 -0
- package/assets/init/.agents/skills/earos-review/references/challenge-patterns.md +163 -0
- package/assets/init/.agents/skills/earos-review/references/output-template.md +180 -0
- package/assets/init/.agents/skills/earos-template-fill/SKILL.md +177 -0
- package/assets/init/.agents/skills/earos-template-fill/references/evidence-writing-guide.md +186 -0
- package/assets/init/.agents/skills/earos-template-fill/references/section-rubric-mapping.md +200 -0
- package/assets/init/.agents/skills/earos-validate/SKILL.md +113 -0
- package/assets/init/.agents/skills/earos-validate/references/fix-patterns.md +281 -0
- package/assets/init/.agents/skills/earos-validate/references/validation-checks.md +287 -0
- package/assets/init/.claude/CLAUDE.md +4 -0
- package/assets/init/AGENTS.md +293 -0
- package/assets/init/CLAUDE.md +635 -0
- package/assets/init/README.md +507 -0
- package/assets/init/calibration/gold-set/.gitkeep +0 -0
- package/assets/init/calibration/results/.gitkeep +0 -0
- package/assets/init/core/core-meta-rubric.yaml +643 -0
- package/assets/init/docs/consistency-report.md +325 -0
- package/assets/init/docs/getting-started.md +194 -0
- package/assets/init/docs/profile-authoring-guide.md +51 -0
- package/assets/init/docs/terminology.md +126 -0
- package/assets/init/earos.manifest.yaml +104 -0
- package/assets/init/evaluations/.gitkeep +0 -0
- package/assets/init/examples/aws-event-driven-order-processing/artifact.yaml +2056 -0
- package/assets/init/examples/aws-event-driven-order-processing/evaluation.yaml +973 -0
- package/assets/init/examples/aws-event-driven-order-processing/report.md +244 -0
- package/assets/init/examples/example-solution-architecture.evaluation.yaml +136 -0
- package/assets/init/examples/multi-cloud-data-analytics/artifact.yaml +715 -0
- package/assets/init/overlays/data-governance.yaml +94 -0
- package/assets/init/overlays/regulatory.yaml +154 -0
- package/assets/init/overlays/security.yaml +92 -0
- package/assets/init/profiles/adr.yaml +225 -0
- package/assets/init/profiles/capability-map.yaml +223 -0
- package/assets/init/profiles/reference-architecture.yaml +426 -0
- package/assets/init/profiles/roadmap.yaml +205 -0
- package/assets/init/profiles/solution-architecture.yaml +227 -0
- package/assets/init/research/architecture-assessment-rubrics-research.docx +0 -0
- package/assets/init/research/architecture-assessment-rubrics-research.md +566 -0
- package/assets/init/research/reference-architecture-research.md +751 -0
- package/assets/init/standard/EAROS.md +1426 -0
- package/assets/init/standard/schemas/artifact.schema.json +1295 -0
- package/assets/init/standard/schemas/artifact.uischema.json +65 -0
- package/assets/init/standard/schemas/evaluation.schema.json +284 -0
- package/assets/init/standard/schemas/rubric.schema.json +383 -0
- package/assets/init/templates/evaluation-record.template.yaml +58 -0
- package/assets/init/templates/new-profile.template.yaml +65 -0
- package/bin.js +188 -0
- package/dist/assets/_basePickBy-BVu6YmSW.js +1 -0
- package/dist/assets/_baseUniq-CWRzQDz_.js +1 -0
- package/dist/assets/arc-CyDBhtDM.js +1 -0
- package/dist/assets/architectureDiagram-2XIMDMQ5-BH6O4dvN.js +36 -0
- package/dist/assets/blockDiagram-WCTKOSBZ-2xmwdjpg.js +132 -0
- package/dist/assets/c4Diagram-IC4MRINW-BNmPRFJF.js +10 -0
- package/dist/assets/channel-CiySTNoJ.js +1 -0
- package/dist/assets/chunk-4BX2VUAB-DGQTvirp.js +1 -0
- package/dist/assets/chunk-55IACEB6-DNMAQAC_.js +1 -0
- package/dist/assets/chunk-FMBD7UC4-BJbVTQ5o.js +15 -0
- package/dist/assets/chunk-JSJVCQXG-BCxUL74A.js +1 -0
- package/dist/assets/chunk-KX2RTZJC-H7wWZOfz.js +1 -0
- package/dist/assets/chunk-NQ4KR5QH-BK4RlTQF.js +220 -0
- package/dist/assets/chunk-QZHKN3VN-0chxDV5g.js +1 -0
- package/dist/assets/chunk-WL4C6EOR-DexfQ-AV.js +189 -0
- package/dist/assets/classDiagram-VBA2DB6C-D7luWJQn.js +1 -0
- package/dist/assets/classDiagram-v2-RAHNMMFH-D7luWJQn.js +1 -0
- package/dist/assets/clone-ylgRbd3D.js +1 -0
- package/dist/assets/cose-bilkent-S5V4N54A-DS2IOCfZ.js +1 -0
- package/dist/assets/cytoscape.esm-CyJtwmzi.js +331 -0
- package/dist/assets/dagre-KLK3FWXG-BbSoTTa3.js +4 -0
- package/dist/assets/defaultLocale-DX6XiGOO.js +1 -0
- package/dist/assets/diagram-E7M64L7V-C9TvYgv0.js +24 -0
- package/dist/assets/diagram-IFDJBPK2-DowUMWrg.js +43 -0
- package/dist/assets/diagram-P4PSJMXO-BL6nrnQF.js +24 -0
- package/dist/assets/erDiagram-INFDFZHY-rXPRl8VM.js +70 -0
- package/dist/assets/flowDiagram-PKNHOUZH-DBRM99-W.js +162 -0
- package/dist/assets/ganttDiagram-A5KZAMGK-INcWFsBT.js +292 -0
- package/dist/assets/gitGraphDiagram-K3NZZRJ6-DMwpfE91.js +65 -0
- package/dist/assets/graph-DLQn37b-.js +1 -0
- package/dist/assets/index-BFFITMT8.js +650 -0
- package/dist/assets/index-H7f6VTz1.css +1 -0
- package/dist/assets/infoDiagram-LFFYTUFH-B0f4TWRM.js +2 -0
- package/dist/assets/init-Gi6I4Gst.js +1 -0
- package/dist/assets/ishikawaDiagram-PHBUUO56-CsU6XimZ.js +70 -0
- package/dist/assets/journeyDiagram-4ABVD52K-CQ7ibNib.js +139 -0
- package/dist/assets/kanban-definition-K7BYSVSG-DzEN7THt.js +89 -0
- package/dist/assets/katex-B1X10hvy.js +261 -0
- package/dist/assets/layout-C0dvb42R.js +1 -0
- package/dist/assets/linear-j4a8mGj7.js +1 -0
- package/dist/assets/mindmap-definition-YRQLILUH-DP8iEuCf.js +68 -0
- package/dist/assets/ordinal-Cboi1Yqb.js +1 -0
- package/dist/assets/pieDiagram-SKSYHLDU-BpIAXgAm.js +30 -0
- package/dist/assets/quadrantDiagram-337W2JSQ-DrpXn5Eg.js +7 -0
- package/dist/assets/requirementDiagram-Z7DCOOCP-Bg7EwHlG.js +73 -0
- package/dist/assets/sankeyDiagram-WA2Y5GQK-BWagRs1F.js +10 -0
- package/dist/assets/sequenceDiagram-2WXFIKYE-q5jwhivG.js +145 -0
- package/dist/assets/stateDiagram-RAJIS63D-B_J9pE-2.js +1 -0
- package/dist/assets/stateDiagram-v2-FVOUBMTO-Q_1GcybB.js +1 -0
- package/dist/assets/timeline-definition-YZTLITO2-dv0jgQ0z.js +61 -0
- package/dist/assets/treemap-KZPCXAKY-Dt1dkIE7.js +162 -0
- package/dist/assets/vennDiagram-LZ73GAT5-BdO5RgRZ.js +34 -0
- package/dist/assets/xychartDiagram-JWTSCODW-CpDVe-8v.js +7 -0
- package/dist/index.html +23 -0
- package/export-docx.js +1583 -0
- package/init.js +353 -0
- package/manifest-cli.mjs +207 -0
- package/package.json +83 -0
- package/schemas/artifact.schema.json +1295 -0
- package/schemas/artifact.uischema.json +65 -0
- package/schemas/evaluation.schema.json +284 -0
- package/schemas/rubric.schema.json +383 -0
- package/serve.js +238 -0
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: earos-report
|
|
3
|
+
description: "Generate executive reports from EAROS evaluation records. Triggers when the user wants to generate a report, create a summary, produce an executive view, aggregate multiple evaluations, show trends, or says \"generate a report\", \"create an executive summary\", \"summarize these evaluations\", \"show me the portfolio status\", \"create a dashboard view\", \"what is the overall quality of our architecture portfolio\", or \"produce an EAROS report\"."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# EAROS Report — Executive Report Generator
|
|
7
|
+
|
|
8
|
+
You generate executive-quality reports from one or more EAROS evaluation records. Reports are audience-aware, status-accurate, and action-oriented. A report that softens a Reject to avoid uncomfortable conversations is worse than no report.
|
|
9
|
+
|
|
10
|
+
**Why this matters:** An evaluation record is a technical artifact — a YAML file full of criterion scores. Decision-makers need a different format: one that surfaces gate failures prominently, provides traffic-light status, ranks actions by impact, and for portfolios, identifies systemic patterns. The report is where evaluation value is realised.
|
|
11
|
+
|
|
12
|
+
## What You Need Before Starting
|
|
13
|
+
|
|
14
|
+
Ask the user three questions if not already clear:
|
|
15
|
+
|
|
16
|
+
1. **Scope** — one evaluation record, or multiple (portfolio)?
|
|
17
|
+
2. **Records location** — specific file path, or scan `evaluations/` and `examples/`?
|
|
18
|
+
3. **Audience** — Architecture Board | Executive | Delivery team | Audit?
|
|
19
|
+
|
|
20
|
+
The audience changes emphasis: Audit audiences get an evidence quality section. Executive audiences get fewer criterion details and more action focus.
|
|
21
|
+
|
|
22
|
+
## Step 1 — Locate and Load Records
|
|
23
|
+
|
|
24
|
+
Scan the specified paths. For each evaluation record:
|
|
25
|
+
- Verify required fields: `evaluation_id`, `artifact_ref`, `status`, `overall_score`, `criterion_results`, `dimension_scores`
|
|
26
|
+
- Note any structural issues — flag them but do not block reporting
|
|
27
|
+
- Extract: status, overall_score, gate_failures, dimension_scores, top_actions, evaluation_date, evaluators
|
|
28
|
+
|
|
29
|
+
## Step 2 — Select Report Mode
|
|
30
|
+
|
|
31
|
+
- **1 record** → Single-artifact report (read `references/single-artifact-template.md`)
|
|
32
|
+
- **2+ records** → Portfolio report (read `references/portfolio-template.md`)
|
|
33
|
+
|
|
34
|
+
## Step 3 — Build the Report
|
|
35
|
+
|
|
36
|
+
Read the appropriate template reference file now. Populate it from the evaluation data.
|
|
37
|
+
|
|
38
|
+
**Traffic light rules (non-negotiable):**
|
|
39
|
+
- Pass ≥ 3.2, no gate failures, no dimension < 2.0 → 🟢
|
|
40
|
+
- Conditional Pass 2.4–3.19, no critical gate failures → 🟡
|
|
41
|
+
- Rework Required < 2.4, or dimension < 2.0 → 🟠
|
|
42
|
+
- Reject: any critical gate failure → 🔴
|
|
43
|
+
- Not Reviewable: insufficient evidence → ⚫
|
|
44
|
+
|
|
45
|
+
**Gate failures go first.** In any report format, gate failures are the most important finding. They appear in their own section immediately after the status header — never buried in a details table.
|
|
46
|
+
|
|
47
|
+
**Actions must be specific.** "Improve traceability" is not an action.
|
|
48
|
+
"Add a traceability matrix linking each business driver to the design sections that implement it (TRC-01)" is an action.
|
|
49
|
+
|
|
50
|
+
## Step 4 — Audience Adjustments
|
|
51
|
+
|
|
52
|
+
| Audience | Emphasis |
|
|
53
|
+
|----------|----------|
|
|
54
|
+
| Architecture Board | Full criterion table, gate analysis, evidence quality |
|
|
55
|
+
| Executive | Status + gate failures + top 5 actions only |
|
|
56
|
+
| Delivery team | Criterion detail + specific remediation steps |
|
|
57
|
+
| Audit | Evidence quality section (observed / inferred / external breakdown) |
|
|
58
|
+
|
|
59
|
+
For Audit audiences, add an evidence quality section from the template.
|
|
60
|
+
|
|
61
|
+
## Step 5 — Save and Confirm
|
|
62
|
+
|
|
63
|
+
Save as:
|
|
64
|
+
- Single artifact: `evaluations/[evaluation_id]-report.md`
|
|
65
|
+
- Portfolio: `evaluations/portfolio-report-[YYYY-MM-DD].md`
|
|
66
|
+
|
|
67
|
+
Confirm the file path with the user. Offer Word/PDF conversion via the docx or pdf skills if needed.
|
|
68
|
+
|
|
69
|
+
## Non-Negotiable Rules
|
|
70
|
+
|
|
71
|
+
1. **Traffic lights must be accurate.** Do not soften Reject to Conditional Pass to ease the conversation.
|
|
72
|
+
2. **Gate failures are prominent.** Never bury them in a criterion details table.
|
|
73
|
+
3. **Evidence quality is separate from score.** An artifact can score 3.0 with mostly inferred evidence — this matters, especially for audit audiences.
|
|
74
|
+
4. **No trends without data.** Don't synthesise trend lines from a single data point per artifact.
|
|
75
|
+
5. **Portfolio health requires honest math.** Compute pass rates from actual statuses — don't round up.
|
|
76
|
+
|
|
77
|
+
## When to Read References
|
|
78
|
+
|
|
79
|
+
| When | Read |
|
|
80
|
+
|------|------|
|
|
81
|
+
| Generating a single-artifact report | `references/single-artifact-template.md` |
|
|
82
|
+
| Generating a portfolio or trend report | `references/portfolio-template.md` |
|
|
83
|
+
| Unsure about traffic light assignment | `references/single-artifact-template.md#status` |
|
|
84
|
+
| Building the gate failure section | Either template — gate section is consistent |
|
|
85
|
+
| Audit audience — evidence quality section | `references/single-artifact-template.md#evidence-quality` |
|
|
@@ -0,0 +1,181 @@
|
|
|
1
|
+
# Portfolio Report Template
|
|
2
|
+
|
|
3
|
+
Use this template when generating a report covering 2 or more EAROS evaluation records. Populate from all evaluation records. Trend analysis requires at least 2 evaluation dates per artifact.
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
```markdown
|
|
8
|
+
# EAROS Portfolio Assessment Report
|
|
9
|
+
|
|
10
|
+
**Report Date:** [today's date]
|
|
11
|
+
**Artifacts Assessed:** [N]
|
|
12
|
+
**Period Covered:** [earliest evaluation_date] to [latest evaluation_date]
|
|
13
|
+
**Report Generated By:** [agent/human]
|
|
14
|
+
**Rubrics Applied:** [list unique rubric_ids across all evaluations]
|
|
15
|
+
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
## Portfolio Health Dashboard
|
|
19
|
+
|
|
20
|
+
| Status | Count | % of portfolio |
|
|
21
|
+
|--------|-------|---------------|
|
|
22
|
+
| 🟢 Pass | [N] | [%] |
|
|
23
|
+
| 🟡 Conditional Pass | [N] | [%] |
|
|
24
|
+
| 🟠 Rework Required | [N] | [%] |
|
|
25
|
+
| 🔴 Reject | [N] | [%] |
|
|
26
|
+
| ⚫ Not Reviewable | [N] | [%] |
|
|
27
|
+
| **Total** | **[N]** | **100%** |
|
|
28
|
+
|
|
29
|
+
**Portfolio Health:** [classification below]
|
|
30
|
+
- **Good** (≥70% Pass or Conditional Pass): Architecture governance is producing acceptable quality
|
|
31
|
+
- **Needs Attention** (40–69% Pass or Conditional Pass): Systemic quality issues evident; intervention recommended
|
|
32
|
+
- **Poor** (<40% Pass or Conditional Pass): Governance process or artifact quality requires immediate attention
|
|
33
|
+
|
|
34
|
+
**Summary statement:** [1–2 sentence overall assessment]
|
|
35
|
+
|
|
36
|
+
---
|
|
37
|
+
|
|
38
|
+
## Artifact Summary
|
|
39
|
+
|
|
40
|
+
| Artifact | Type | Eval Date | Status | Score | Gate Failures |
|
|
41
|
+
|----------|------|-----------|--------|-------|--------------|
|
|
42
|
+
| [title] | [type] | [date] | [🟢/🟡/🟠/🔴/⚫] [status] | [X.X] | [N or None] |
|
|
43
|
+
| ... | | | | | |
|
|
44
|
+
|
|
45
|
+
**Sort order:** By status (Reject first, then Rework, then Conditional, then Pass), then by score ascending within each group. This surfaces the most urgent issues first.
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## Dimension Analysis
|
|
50
|
+
|
|
51
|
+
*Average scores across all artifacts, by dimension.*
|
|
52
|
+
|
|
53
|
+
| Dimension | Avg Score | Lowest | Highest | Below 2.0 Count |
|
|
54
|
+
|-----------|-----------|--------|---------|-----------------|
|
|
55
|
+
| [dim name] | [avg] | [min — artifact] | [max — artifact] | [N artifacts] |
|
|
56
|
+
| ... | | | | |
|
|
57
|
+
|
|
58
|
+
**Systemic weaknesses** (dimensions with average < 2.5 across portfolio):
|
|
59
|
+
[List each weak dimension and note which artifact types or teams it affects]
|
|
60
|
+
|
|
61
|
+
**Portfolio strengths** (dimensions with average ≥ 3.2 across portfolio):
|
|
62
|
+
[List each strong dimension]
|
|
63
|
+
|
|
64
|
+
---
|
|
65
|
+
|
|
66
|
+
## Gate Failure Analysis
|
|
67
|
+
|
|
68
|
+
*Which gate criteria are failing and how often.*
|
|
69
|
+
|
|
70
|
+
[If no gate failures across portfolio:]
|
|
71
|
+
> No gate failures across the portfolio.
|
|
72
|
+
|
|
73
|
+
[If gate failures present:]
|
|
74
|
+
|
|
75
|
+
| Criterion | Gate Severity | Artifacts Failed | Failure Rate | Status Effect |
|
|
76
|
+
|-----------|--------------|-----------------|-------------|---------------|
|
|
77
|
+
| [ID] | [critical/major] | [N] ([list artifact names]) | [%] | [reject/cap at conditional_pass] |
|
|
78
|
+
| ... | | | | |
|
|
79
|
+
|
|
80
|
+
**Systemic gate failures** (criterion failing in ≥30% of artifacts):
|
|
81
|
+
[Analysis of why the same gate keeps failing — is it a rubric clarity issue, an authoring practice issue, or a genuine governance gap?]
|
|
82
|
+
|
|
83
|
+
> If a gate criterion fails across multiple artifacts from different teams, it suggests a shared knowledge gap rather than individual team failures. Consider: training, template improvements, or rubric clarification.
|
|
84
|
+
|
|
85
|
+
---
|
|
86
|
+
|
|
87
|
+
## Trend Analysis
|
|
88
|
+
|
|
89
|
+
*Only include if 2+ evaluations exist for any artifact, or the portfolio spans 2+ time periods.*
|
|
90
|
+
|
|
91
|
+
| Artifact | Previous Score | Current Score | Change | Status Change |
|
|
92
|
+
|----------|---------------|---------------|--------|---------------|
|
|
93
|
+
| [title] | [X.X] ([date]) | [X.X] ([date]) | [+/-X.X] | [same/improved/declined] |
|
|
94
|
+
| ... | | | | |
|
|
95
|
+
|
|
96
|
+
**Portfolio trend:** [Improving / Stable / Declining]
|
|
97
|
+
- Average score change: [+/-X.X] since [period]
|
|
98
|
+
- Pass rate change: [+/-X%] since [period]
|
|
99
|
+
|
|
100
|
+
**Notable changes:**
|
|
101
|
+
- [Artifact that improved most significantly and why]
|
|
102
|
+
- [Artifact that declined most significantly and why]
|
|
103
|
+
|
|
104
|
+
---
|
|
105
|
+
|
|
106
|
+
## Dimension Trends
|
|
107
|
+
|
|
108
|
+
*Only include if multiple evaluation periods exist.*
|
|
109
|
+
|
|
110
|
+
| Dimension | [Period 1] Avg | [Period 2] Avg | Trend |
|
|
111
|
+
|-----------|---------------|---------------|-------|
|
|
112
|
+
| [dim name] | [X.X] | [X.X] | [↑ improving / → stable / ↓ declining] |
|
|
113
|
+
|
|
114
|
+
---
|
|
115
|
+
|
|
116
|
+
## Evidence Quality Summary
|
|
117
|
+
|
|
118
|
+
*Aggregate evidence quality across all evaluations.*
|
|
119
|
+
|
|
120
|
+
| Evidence Class | Total Criteria | % of scored criteria |
|
|
121
|
+
|---------------|---------------|---------------------|
|
|
122
|
+
| Observed | [N] | [%] |
|
|
123
|
+
| Inferred | [N] | [%] |
|
|
124
|
+
| External | [N] | [%] |
|
|
125
|
+
|
|
126
|
+
**Portfolio evidence reliability:** [Strong (>80% observed) / Moderate / Low]
|
|
127
|
+
|
|
128
|
+
[If evidence reliability is Low:]
|
|
129
|
+
> ⚠️ The portfolio has high inferred evidence ratios. This may indicate that artifacts are not making content explicit enough for assessors to find direct evidence. Recommend sharing the EAROS evidence writing guide with artifact authors.
|
|
130
|
+
|
|
131
|
+
---
|
|
132
|
+
|
|
133
|
+
## Portfolio Recommendations
|
|
134
|
+
|
|
135
|
+
*Top systemic improvements. Focus on issues affecting multiple artifacts.*
|
|
136
|
+
|
|
137
|
+
| # | Recommendation | Artifacts Affected | Expected Impact |
|
|
138
|
+
|---|---------------|-------------------|-----------------|
|
|
139
|
+
| 1 | [Specific, actionable recommendation] | [N artifacts] | [Pass rate / score improvement] |
|
|
140
|
+
| 2 | [Recommendation] | [N artifacts] | [Impact] |
|
|
141
|
+
| ... | | | |
|
|
142
|
+
|
|
143
|
+
**Most impactful single action:** [The one change that would most improve portfolio health]
|
|
144
|
+
|
|
145
|
+
---
|
|
146
|
+
|
|
147
|
+
## Individual Artifact Summaries
|
|
148
|
+
|
|
149
|
+
*3-line summary per artifact: status, top strength, top gap.*
|
|
150
|
+
|
|
151
|
+
**[Artifact Title]** — 🟢/🟡/🟠/🔴/⚫ [Status] | Score: [X.X]
|
|
152
|
+
- Strength: [Specific high-scoring criterion and evidence]
|
|
153
|
+
- Gap: [Most critical low-scoring or gate criterion]
|
|
154
|
+
- Action: [Single most important improvement]
|
|
155
|
+
|
|
156
|
+
[Repeat for each artifact]
|
|
157
|
+
|
|
158
|
+
---
|
|
159
|
+
|
|
160
|
+
## Appendix: Criterion Heat Map
|
|
161
|
+
|
|
162
|
+
*Score distribution across all artifacts by criterion.*
|
|
163
|
+
|
|
164
|
+
| Criterion | Avg | [Art 1] | [Art 2] | [Art 3] | ... |
|
|
165
|
+
|-----------|-----|---------|---------|---------|-----|
|
|
166
|
+
| [ID] | [avg] | [score] | [score] | [score] | |
|
|
167
|
+
| ... | | | | | |
|
|
168
|
+
|
|
169
|
+
**Colour key:** 🟢 Score ≥3 | 🟡 Score 2 | 🟠 Score 1 | 🔴 Score 0 | — N/A
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
---
|
|
173
|
+
|
|
174
|
+
## Template Notes
|
|
175
|
+
|
|
176
|
+
- **Portfolio health classification** is computed from actual statuses — do not estimate
|
|
177
|
+
- **Systemic gate failures** are the most important finding in a portfolio report — always investigate and explain
|
|
178
|
+
- **Trend analysis** requires dates — never synthesise trends from a single evaluation per artifact
|
|
179
|
+
- **Individual summaries** should be 3 lines maximum — detail belongs in per-artifact reports
|
|
180
|
+
- **Recommendations** must be systemic (affecting multiple artifacts) — individual artifact actions belong in the single-artifact report
|
|
181
|
+
- Save as: `evaluations/portfolio-report-[YYYY-MM-DD].md`
|
|
@@ -0,0 +1,168 @@
|
|
|
1
|
+
# Single-Artifact Report Template
|
|
2
|
+
|
|
3
|
+
Use this template when generating a report for one EAROS evaluation record. Populate all fields from the evaluation record. Do not omit sections — even if empty, they must appear to confirm they were checked.
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
```markdown
|
|
8
|
+
# EAROS Assessment Report
|
|
9
|
+
|
|
10
|
+
**Artifact:** [artifact_ref.title]
|
|
11
|
+
**Artifact Type:** [artifact_ref.artifact_type]
|
|
12
|
+
**Owner:** [artifact_ref.owner or "Not specified"]
|
|
13
|
+
**Version:** [artifact_ref.version or "Not specified"]
|
|
14
|
+
**Evaluation Date:** [evaluation_date]
|
|
15
|
+
**Evaluated By:** [evaluators list — names and modes (agent/human/hybrid)]
|
|
16
|
+
**Rubric Applied:** [rubric_id, including profile and any overlays]
|
|
17
|
+
|
|
18
|
+
---
|
|
19
|
+
|
|
20
|
+
## Status {#status}
|
|
21
|
+
|
|
22
|
+
| | |
|
|
23
|
+
|---|---|
|
|
24
|
+
| **Overall Status** | [TRAFFIC LIGHT] [STATUS LABEL] |
|
|
25
|
+
| **Overall Score** | [overall_score] / 4.0 |
|
|
26
|
+
| **Gate Failures** | [count or "None"] |
|
|
27
|
+
|
|
28
|
+
**Traffic light assignment:**
|
|
29
|
+
- 🟢 **Pass** — No gate failures. Overall ≥ 3.2. No dimension < 2.0.
|
|
30
|
+
- 🟡 **Conditional Pass** — No critical gate failures. Overall 2.4–3.19. Named remediation items required.
|
|
31
|
+
- 🟠 **Rework Required** — Overall < 2.4, or repeated weak dimensions, or insufficient evidence.
|
|
32
|
+
- 🔴 **Reject** — Critical gate failure. Cannot be approved regardless of overall score.
|
|
33
|
+
- ⚫ **Not Reviewable** — Evidence too incomplete to score key criteria.
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## Gate Failures
|
|
38
|
+
|
|
39
|
+
[If gate_failures list is empty:]
|
|
40
|
+
> No gate failures — all gate criteria cleared.
|
|
41
|
+
|
|
42
|
+
[If gate_failures present:]
|
|
43
|
+
> ⚠️ Gate failures override the weighted average. The status below reflects the gate outcome, not just the score.
|
|
44
|
+
|
|
45
|
+
| Criterion | Gate Severity | Score | Effect |
|
|
46
|
+
|-----------|--------------|-------|--------|
|
|
47
|
+
| [criterion_id] | [critical/major] | [score] | [failure_effect from rubric] |
|
|
48
|
+
|
|
49
|
+
---
|
|
50
|
+
|
|
51
|
+
## Dimension Scorecard
|
|
52
|
+
|
|
53
|
+
| Dimension | Score | Weight | Status |
|
|
54
|
+
|-----------|-------|--------|--------|
|
|
55
|
+
| [dimension_name] | [score] / 4.0 | [weight] | [🟢 ≥3.2 / 🟡 2.4–3.19 / 🟠 <2.4] |
|
|
56
|
+
| ... | | | |
|
|
57
|
+
| **Weighted Overall** | **[overall_score]** | | [status label] |
|
|
58
|
+
|
|
59
|
+
**Dimension floor check:** [Pass / Fail — any dimension < 2.0?]
|
|
60
|
+
If any dimension < 2.0: "⚠️ [Dimension name] scored [score] — below the 2.0 floor required for Pass status."
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
## Key Findings
|
|
65
|
+
|
|
66
|
+
### Strengths
|
|
67
|
+
[3 bullet points. For each: criterion ID, score, and the specific evidence that earns it. One sentence per bullet.]
|
|
68
|
+
|
|
69
|
+
- **[Criterion ID] — [criterion name]** (score [X]): [Specific evidence that demonstrates strength]
|
|
70
|
+
- **[Criterion ID] — [criterion name]** (score [X]): [Specific evidence]
|
|
71
|
+
- **[Criterion ID] — [criterion name]** (score [X]): [Specific evidence]
|
|
72
|
+
|
|
73
|
+
### Critical Gaps
|
|
74
|
+
[3–5 bullet points. For each: criterion ID, score, and what specifically is missing. Gate criteria listed first.]
|
|
75
|
+
|
|
76
|
+
- **[Criterion ID] — [criterion name]** (score [X]) [⚠️ GATE]: [What is missing and why it matters]
|
|
77
|
+
- **[Criterion ID] — [criterion name]** (score [X]): [What is missing]
|
|
78
|
+
- ...
|
|
79
|
+
|
|
80
|
+
---
|
|
81
|
+
|
|
82
|
+
## Recommended Actions
|
|
83
|
+
|
|
84
|
+
Actions are ordered by priority. Gate-related actions must be addressed before resubmission.
|
|
85
|
+
|
|
86
|
+
| # | Action | Criterion | Priority | Suggested Owner |
|
|
87
|
+
|---|--------|-----------|----------|-----------------|
|
|
88
|
+
| 1 | [Specific, verb-first action] | [ID] | Critical — Gate | [role] |
|
|
89
|
+
| 2 | [Specific action] | [ID] | High | [role] |
|
|
90
|
+
| 3 | [Specific action] | [ID] | Medium | [role] |
|
|
91
|
+
| ... | | | | |
|
|
92
|
+
|
|
93
|
+
**Action quality standard:** "Add a scope section listing in-scope and out-of-scope items explicitly, including at least 3 named assumptions (SCP-01)" is a good action. "Improve scope" is not.
|
|
94
|
+
|
|
95
|
+
---
|
|
96
|
+
|
|
97
|
+
## Detailed Criterion Results
|
|
98
|
+
|
|
99
|
+
| Criterion | Dimension | Score | Conf. | Evidence Class | N/A Reason |
|
|
100
|
+
|-----------|-----------|-------|-------|---------------|------------|
|
|
101
|
+
| [ID] | [dim] | [0-4/N/A] | [H/M/L] | [obs/inf/ext] | [if N/A: reason] |
|
|
102
|
+
| ... | | | | | |
|
|
103
|
+
|
|
104
|
+
---
|
|
105
|
+
|
|
106
|
+
## Evidence Quality Summary {#evidence-quality}
|
|
107
|
+
|
|
108
|
+
*Include for Architecture Board and Audit audiences. Optional for delivery teams.*
|
|
109
|
+
|
|
110
|
+
| Evidence Type | Count | % of scored criteria |
|
|
111
|
+
|--------------|-------|---------------------|
|
|
112
|
+
| Observed (direct quote or section reference) | [N] | [%] |
|
|
113
|
+
| Inferred (reasonable interpretation) | [N] | [%] |
|
|
114
|
+
| External (based on standard or policy) | [N] | [%] |
|
|
115
|
+
| N/A (criterion genuinely not applicable) | [N] | [%] |
|
|
116
|
+
|
|
117
|
+
**Evidence reliability:** [Strong (>80% observed) / Moderate (50–80%) / Low (<50%)]
|
|
118
|
+
|
|
119
|
+
[If low confidence count > 20%:]
|
|
120
|
+
> ⚠️ [N] criteria scored with low confidence. Human reviewer judgment recommended for these before relying on this assessment for governance decisions.
|
|
121
|
+
|
|
122
|
+
---
|
|
123
|
+
|
|
124
|
+
## Assessment Notes
|
|
125
|
+
|
|
126
|
+
[narrative_summary from the evaluation record, if present. Otherwise: synthesise a 2–3 paragraph summary covering:]
|
|
127
|
+
1. What the artifact does well and why
|
|
128
|
+
2. The primary gaps and their downstream consequences
|
|
129
|
+
3. The recommended path forward (resubmit after addressing X, escalate to Y, etc.)
|
|
130
|
+
|
|
131
|
+
---
|
|
132
|
+
|
|
133
|
+
## Evaluation Metadata
|
|
134
|
+
|
|
135
|
+
| Field | Value |
|
|
136
|
+
|-------|-------|
|
|
137
|
+
| Evaluation ID | [evaluation_id] |
|
|
138
|
+
| Rubric version | [rubric version] |
|
|
139
|
+
| Evaluation mode | [agent / human / hybrid] |
|
|
140
|
+
| Calibration status | [calibrated / uncalibrated — if uncalibrated, note this] |
|
|
141
|
+
| Challenger review | [completed / not performed] |
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
---
|
|
145
|
+
|
|
146
|
+
## Challenger Score {#challenger-score}
|
|
147
|
+
|
|
148
|
+
If a challenger review was performed (earos-review skill), add this section:
|
|
149
|
+
|
|
150
|
+
```markdown
|
|
151
|
+
## Challenger Review
|
|
152
|
+
|
|
153
|
+
**Challenger Status:** [Concur with primary / Revised to: (status)]
|
|
154
|
+
**Challenger Score:** [Concur / (score) — differs from primary by (delta)]
|
|
155
|
+
**Challenging findings:** [N] BLOCKER / [N] MAJOR / [N] MINOR
|
|
156
|
+
|
|
157
|
+
[For each BLOCKER or MAJOR finding: criterion ID, primary score, challenger score, reason]
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
---
|
|
161
|
+
|
|
162
|
+
## Formatting Notes
|
|
163
|
+
|
|
164
|
+
- Gate failures must appear prominently — in their own section immediately after the status, not buried in the criterion table
|
|
165
|
+
- Traffic light emoji must be accurate — do not use 🟡 when the status is Reject
|
|
166
|
+
- Actions must be specific — verb first, criterion reference included
|
|
167
|
+
- Scores like "2.8" mean the weighted average is 2.8, not that each criterion scored 2.8
|
|
168
|
+
- For scores of N/A: the criterion is excluded from the denominator — document why
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: earos-review
|
|
3
|
+
description: "Challenge and peer-review an existing EAROS evaluation record. Use this skill whenever someone wants to audit, second-opinion, or challenge a completed evaluation. Triggers on \"check this evaluation\", \"challenge these scores\", \"review the assessment\", \"second opinion on this\", \"audit this EAROS record\", \"are these scores right\", \"was this evaluation fair\", \"over-scored\", \"too generous\", \"missed a gate failure\", \"verify this assessment\", \"quality check this evaluation\", or any request to validate evaluation quality. Also triggers when a YAML evaluation record is provided alongside the original artifact and the user asks for a quality check. This is distinct from earos-assess (which runs a fresh evaluation) — earos-review audits an existing one."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# EAROS Review (Challenger) Skill
|
|
7
|
+
|
|
8
|
+
You are the challenger evaluator. Your job is not to re-evaluate the artifact from scratch — it is to audit the evaluation record itself. You check whether the primary evaluator's scores are supported by the evidence they cited, consistent with the rubric's level descriptors, and free from the systematic biases that plague architecture assessment.
|
|
9
|
+
|
|
10
|
+
**Why this matters:** The most common failure modes in EAROS evaluation are not random errors — they are systematic: over-scoring well-written prose, misclassifying inferred evidence as observed, and missing gate failures that change the final status. A challenger who knows what to look for catches these reliably. Without a challenge pass, inflated evaluations reach governance boards unchecked.
|
|
11
|
+
|
|
12
|
+
**Before running Phase 2:** Read `references/challenge-patterns.md`. It describes the 5 systemic failure modes with detection guidance and examples.
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
## Inputs Required
|
|
17
|
+
|
|
18
|
+
You need three things. If any are missing, ask before proceeding:
|
|
19
|
+
|
|
20
|
+
1. **The evaluation record** — a YAML file (usually in `evaluations/` or `examples/`)
|
|
21
|
+
2. **The original artifact** — the document or design that was evaluated
|
|
22
|
+
3. **The rubric files** — identified by `rubric_id` in the evaluation record; load from `core/`, `profiles/`, `overlays/`
|
|
23
|
+
|
|
24
|
+
Also load `standard/schemas/evaluation.schema.json` for structural validation.
|
|
25
|
+
|
|
26
|
+
---
|
|
27
|
+
|
|
28
|
+
## Phase 1 — Schema and Structural Check
|
|
29
|
+
|
|
30
|
+
*Purpose: catch invisible errors — missing fields, skipped criteria, inconsistent status.*
|
|
31
|
+
|
|
32
|
+
Check that the evaluation record has:
|
|
33
|
+
- [ ] All required fields: `evaluation_id`, `rubric_id`, `artifact_ref`, `evaluation_date`, `evaluators`, `status`, `overall_score`, `criterion_results`
|
|
34
|
+
- [ ] Every criterion from the loaded rubric appears in `criterion_results` — silently skipped criteria are a red flag
|
|
35
|
+
- [ ] `gate_failures` field present (even if empty)
|
|
36
|
+
- [ ] `recommended_actions` present
|
|
37
|
+
- [ ] Each criterion result has: `score`, `judgment_type`, `confidence`, `evidence_refs`, `rationale`
|
|
38
|
+
- [ ] Status is internally consistent: a `pass` status with a `critical` gate failure is an error
|
|
39
|
+
|
|
40
|
+
Flag every structural violation as **[SCHEMA ERROR]** in the output.
|
|
41
|
+
|
|
42
|
+
---
|
|
43
|
+
|
|
44
|
+
## Phase 2 — Evidence Audit
|
|
45
|
+
|
|
46
|
+
*Purpose: determine whether each score is supported by actual artifact content.*
|
|
47
|
+
|
|
48
|
+
> **Read `references/challenge-patterns.md` before this phase.** It contains detection methods for each failure mode with good and bad examples.
|
|
49
|
+
|
|
50
|
+
For each criterion in the evaluation record:
|
|
51
|
+
|
|
52
|
+
**A. Evidence support check**
|
|
53
|
+
- Locate the `evidence_refs` cited in the evaluation
|
|
54
|
+
- Find those sections in the original artifact
|
|
55
|
+
- Does the excerpt actually say what the rationale claims? Watch for paraphrase-creep — where the evaluator's interpretation gets attributed to the artifact
|
|
56
|
+
- Is the `judgment_type` accurate?
|
|
57
|
+
- `observed` requires a direct quote or clearly stated fact
|
|
58
|
+
- If the evaluator inferred it → should be `inferred`
|
|
59
|
+
- If they applied outside knowledge → should be `external`
|
|
60
|
+
|
|
61
|
+
**B. Score calibration check**
|
|
62
|
+
- Read the `scoring_guide` in the rubric for this criterion
|
|
63
|
+
- Does the score match the level descriptor?
|
|
64
|
+
- For scores of 3 or 4: "Is this genuinely well evidenced, or benefit of the doubt?"
|
|
65
|
+
- For scores of 0 or 1: "Did the evaluator search thoroughly?"
|
|
66
|
+
|
|
67
|
+
**C. Gate check**
|
|
68
|
+
- If `gate.enabled: true`: check the score against the gate threshold
|
|
69
|
+
- If the score fails the gate — is it listed in `gate_failures`?
|
|
70
|
+
- If listed as a gate failure — does the status reflect the correct effect?
|
|
71
|
+
|
|
72
|
+
Record your verdict per criterion:
|
|
73
|
+
|
|
74
|
+
```
|
|
75
|
+
criterion_id: [ID]
|
|
76
|
+
primary_score: [from record]
|
|
77
|
+
challenger_verdict: agree | disagree | partial
|
|
78
|
+
challenger_score: [your score if different]
|
|
79
|
+
issue_type: over_scored | under_scored | evidence_unsupported | wrong_evidence_class | gate_missed | none
|
|
80
|
+
challenge_note: "[specific reason citing the rubric level descriptor]"
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## Phase 3 — Systemic Pattern Analysis
|
|
86
|
+
|
|
87
|
+
*Purpose: identify whether the evaluation has a systematic bias, not just isolated errors.*
|
|
88
|
+
|
|
89
|
+
After reviewing all criteria, look for patterns across the full set:
|
|
90
|
+
|
|
91
|
+
1. **Optimistic evidence classification** — multiple criteria marked `observed` where evidence is actually `inferred`
|
|
92
|
+
2. **Generosity bias** — consistently scoring 3 where 2 is more accurate; benefit-of-the-doubt pattern-wide
|
|
93
|
+
3. **Missing evidence anchors** — rationale cites general impressions rather than specific locations
|
|
94
|
+
4. **Gate blindness** — gate criteria failed but not in `gate_failures`, or status doesn't reflect gate effects
|
|
95
|
+
5. **Confidence inflation** — `high` confidence on criteria with thin or inferred evidence
|
|
96
|
+
|
|
97
|
+
> **For examples of each pattern and how to detect them**, see `references/challenge-patterns.md`.
|
|
98
|
+
|
|
99
|
+
---
|
|
100
|
+
|
|
101
|
+
## Phase 4 — Overall Assessment
|
|
102
|
+
|
|
103
|
+
Compute:
|
|
104
|
+
- Criteria agreed / challenged / evidence-quality issues / gate errors
|
|
105
|
+
- Your challenger overall score (if revised scores produce a different weighted average)
|
|
106
|
+
- Your challenger status recommendation (if it differs from the primary)
|
|
107
|
+
|
|
108
|
+
> **Read `references/output-template.md` before writing the report.** It contains the full format with field-by-field guidance.
|
|
109
|
+
|
|
110
|
+
---
|
|
111
|
+
|
|
112
|
+
## Non-Negotiable Rules
|
|
113
|
+
|
|
114
|
+
1. **Don't soften challenges.** If the evidence doesn't support the score, say so clearly and cite the level descriptor.
|
|
115
|
+
2. **Don't re-score without evidence.** If you cannot find support for a different score in the artifact, do not challenge.
|
|
116
|
+
3. **Gate errors are critical findings.** A missed gate failure that changes the status is not a minor issue — flag it prominently.
|
|
117
|
+
4. **The three evaluation types are distinct.** Check whether artifact quality, architectural fitness, and governance fit have been collapsed into a single judgment.
|
|
118
|
+
5. **Reference level descriptors.** Every disagreement must cite the specific descriptor the primary evaluator should have applied.
|
|
119
|
+
|
|
120
|
+
---
|
|
121
|
+
|
|
122
|
+
## When to Read Which Reference File
|
|
123
|
+
|
|
124
|
+
| When | Read |
|
|
125
|
+
|------|------|
|
|
126
|
+
| Before Phase 2 (always) | `references/challenge-patterns.md` |
|
|
127
|
+
| Detecting a specific failure mode | `references/challenge-patterns.md` |
|
|
128
|
+
| Before writing the challenger report | `references/output-template.md` |
|
|
129
|
+
| Unsure whether to challenge a score | `references/challenge-patterns.md#score-calibration` |
|
|
130
|
+
| Computing challenger overall score | `references/output-template.md#challenger-score` |
|