aw-ecc 1.4.31 → 1.4.47
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/.codex/hooks/aw-post-tool-use.sh +8 -2
- package/.codex/hooks/aw-session-start.sh +11 -4
- package/.codex/hooks/aw-stop.sh +8 -2
- package/.codex/hooks/aw-user-prompt-submit.sh +10 -2
- package/.codex/hooks.json +8 -8
- package/.cursor/INSTALL.md +7 -5
- package/.cursor/hooks/adapter.js +41 -4
- package/.cursor/hooks/after-agent-response.js +62 -0
- package/.cursor/hooks/before-submit-prompt.js +7 -1
- package/.cursor/hooks/post-tool-use-failure.js +21 -0
- package/.cursor/hooks/post-tool-use.js +39 -0
- package/.cursor/hooks/shared/aw-phase-definitions.js +53 -0
- package/.cursor/hooks/shared/aw-phase-runner.js +3 -1
- package/.cursor/hooks/subagent-start.js +22 -4
- package/.cursor/hooks/subagent-stop.js +18 -1
- package/.cursor/hooks.json +23 -2
- package/.opencode/package.json +1 -1
- package/AGENTS.md +3 -3
- package/README.md +5 -5
- package/commands/adk.md +52 -0
- package/commands/build.md +22 -9
- package/commands/deploy.md +12 -0
- package/commands/execute.md +9 -0
- package/commands/feature.md +333 -0
- package/commands/investigate.md +18 -5
- package/commands/plan.md +23 -9
- package/commands/publish.md +65 -0
- package/commands/review.md +12 -0
- package/commands/ship.md +12 -0
- package/commands/test.md +12 -0
- package/commands/verify.md +9 -0
- package/hooks/hooks.json +36 -0
- package/manifests/install-components.json +8 -0
- package/manifests/install-modules.json +83 -0
- package/manifests/install-profiles.json +7 -0
- package/package.json +1 -1
- package/scripts/ci/validate-rules.js +51 -0
- package/scripts/cursor-aw-home/hooks.json +23 -2
- package/scripts/cursor-aw-hooks/adapter.js +41 -4
- package/scripts/cursor-aw-hooks/before-submit-prompt.js +7 -1
- package/scripts/hooks/aw-usage-commit-created.js +32 -0
- package/scripts/hooks/aw-usage-post-tool-use-failure.js +56 -0
- package/scripts/hooks/aw-usage-post-tool-use.js +242 -0
- package/scripts/hooks/aw-usage-prompt-submit.js +112 -0
- package/scripts/hooks/aw-usage-session-start.js +48 -0
- package/scripts/hooks/aw-usage-stop.js +182 -0
- package/scripts/hooks/aw-usage-telemetry-send.js +84 -0
- package/scripts/hooks/cost-tracker.js +3 -23
- package/scripts/hooks/shared/aw-phase-definitions.js +53 -0
- package/scripts/hooks/shared/aw-phase-runner.js +3 -1
- package/scripts/lib/aw-hook-contract.js +2 -2
- package/scripts/lib/aw-pricing.js +306 -0
- package/scripts/lib/aw-usage-telemetry.js +472 -0
- package/scripts/lib/codex-hook-config.js +8 -8
- package/scripts/lib/cursor-hook-config.js +25 -10
- package/scripts/lib/install-targets/codex-home.js +7 -0
- package/scripts/lib/install-targets/cursor-project.js +3 -0
- package/scripts/lib/install-targets/helpers.js +20 -3
- package/skills/aw-adk/SKILL.md +317 -0
- package/skills/aw-adk/agents/analyzer.md +113 -0
- package/skills/aw-adk/agents/comparator.md +113 -0
- package/skills/aw-adk/agents/grader.md +115 -0
- package/skills/aw-adk/assets/eval_review.html +76 -0
- package/skills/aw-adk/eval-viewer/generate_review.py +164 -0
- package/skills/aw-adk/eval-viewer/viewer.html +181 -0
- package/skills/aw-adk/evals/eval-colocated-placement.md +84 -0
- package/skills/aw-adk/evals/eval-create-agent.md +90 -0
- package/skills/aw-adk/evals/eval-create-command.md +98 -0
- package/skills/aw-adk/evals/eval-create-eval.md +89 -0
- package/skills/aw-adk/evals/eval-create-rule.md +99 -0
- package/skills/aw-adk/evals/eval-create-skill.md +97 -0
- package/skills/aw-adk/evals/eval-delete-agent.md +79 -0
- package/skills/aw-adk/evals/eval-delete-command.md +89 -0
- package/skills/aw-adk/evals/eval-delete-rule.md +86 -0
- package/skills/aw-adk/evals/eval-delete-skill.md +90 -0
- package/skills/aw-adk/evals/eval-meta-eval-coverage.md +78 -0
- package/skills/aw-adk/evals/eval-meta-eval-determinism.md +81 -0
- package/skills/aw-adk/evals/eval-meta-eval-false-pass.md +81 -0
- package/skills/aw-adk/evals/eval-score-accuracy.md +95 -0
- package/skills/aw-adk/evals/eval-type-redirect.md +68 -0
- package/skills/aw-adk/evals/evals.json +96 -0
- package/skills/aw-adk/references/artifact-wiring.md +162 -0
- package/skills/aw-adk/references/cross-ide-mapping.md +71 -0
- package/skills/aw-adk/references/eval-placement-guide.md +183 -0
- package/skills/aw-adk/references/external-resources.md +75 -0
- package/skills/aw-adk/references/getting-started.md +66 -0
- package/skills/aw-adk/references/registry-structure.md +152 -0
- package/skills/aw-adk/references/rubric-agent.md +36 -0
- package/skills/aw-adk/references/rubric-command.md +36 -0
- package/skills/aw-adk/references/rubric-eval.md +36 -0
- package/skills/aw-adk/references/rubric-meta-eval.md +132 -0
- package/skills/aw-adk/references/rubric-rule.md +36 -0
- package/skills/aw-adk/references/rubric-skill.md +36 -0
- package/skills/aw-adk/references/schemas.md +222 -0
- package/skills/aw-adk/references/template-agent.md +251 -0
- package/skills/aw-adk/references/template-command.md +279 -0
- package/skills/aw-adk/references/template-eval.md +176 -0
- package/skills/aw-adk/references/template-rule.md +119 -0
- package/skills/aw-adk/references/template-skill.md +123 -0
- package/skills/aw-adk/references/type-classifier.md +98 -0
- package/skills/aw-adk/references/writing-good-agents.md +227 -0
- package/skills/aw-adk/references/writing-good-commands.md +258 -0
- package/skills/aw-adk/references/writing-good-evals.md +271 -0
- package/skills/aw-adk/references/writing-good-rules.md +214 -0
- package/skills/aw-adk/references/writing-good-skills.md +159 -0
- package/skills/aw-adk/scripts/aggregate-benchmark.py +190 -0
- package/skills/aw-adk/scripts/lint-artifact.sh +211 -0
- package/skills/aw-adk/scripts/score-artifact.sh +179 -0
- package/skills/aw-adk/scripts/trigger-eval.py +192 -0
- package/skills/aw-build/SKILL.md +19 -2
- package/skills/aw-deploy/SKILL.md +65 -3
- package/skills/aw-design/SKILL.md +156 -0
- package/skills/aw-design/references/highrise-tokens.md +394 -0
- package/skills/aw-design/references/micro-interactions.md +76 -0
- package/skills/aw-design/references/prompt-template.md +160 -0
- package/skills/aw-design/references/quality-checklist.md +70 -0
- package/skills/aw-design/references/self-review.md +497 -0
- package/skills/aw-design/references/stitch-workflow.md +127 -0
- package/skills/aw-feature/SKILL.md +293 -0
- package/skills/aw-investigate/SKILL.md +17 -0
- package/skills/aw-plan/SKILL.md +34 -3
- package/skills/aw-publish/SKILL.md +300 -0
- package/skills/aw-publish/evals/eval-confirmation-gate.md +60 -0
- package/skills/aw-publish/evals/eval-intent-detection.md +111 -0
- package/skills/aw-publish/evals/eval-push-modes.md +67 -0
- package/skills/aw-publish/evals/eval-rules-push.md +60 -0
- package/skills/aw-publish/evals/evals.json +29 -0
- package/skills/aw-publish/references/push-modes.md +38 -0
- package/skills/aw-review/SKILL.md +88 -9
- package/skills/aw-rules-review/SKILL.md +124 -0
- package/skills/aw-rules-review/agents/openai.yaml +3 -0
- package/skills/aw-rules-review/scripts/generate-review-template.mjs +323 -0
- package/skills/aw-ship/SKILL.md +16 -0
- package/skills/aw-spec/SKILL.md +15 -0
- package/skills/aw-tasks/SKILL.md +15 -0
- package/skills/aw-test/SKILL.md +16 -0
- package/skills/aw-yolo/SKILL.md +4 -0
- package/skills/diagnose/SKILL.md +121 -0
- package/skills/diagnose/scripts/hitl-loop.template.sh +41 -0
- package/skills/finish-only-when-green/SKILL.md +265 -0
- package/skills/grill-me/SKILL.md +24 -0
- package/skills/grill-with-docs/SKILL.md +92 -0
- package/skills/grill-with-docs/adr-format.md +47 -0
- package/skills/grill-with-docs/context-format.md +67 -0
- package/skills/improve-codebase-architecture/SKILL.md +75 -0
- package/skills/improve-codebase-architecture/deepening.md +37 -0
- package/skills/improve-codebase-architecture/interface-design.md +44 -0
- package/skills/improve-codebase-architecture/language.md +53 -0
- package/skills/local-ghl-setup-from-screenshot/SKILL.md +538 -0
- package/skills/tdd/SKILL.md +115 -0
- package/skills/tdd/deep-modules.md +33 -0
- package/skills/tdd/interface-design.md +31 -0
- package/skills/tdd/mocking.md +59 -0
- package/skills/tdd/refactoring.md +10 -0
- package/skills/tdd/tests.md +61 -0
- package/skills/to-issues/SKILL.md +62 -0
- package/skills/to-prd/SKILL.md +75 -0
- package/skills/using-aw-skills/SKILL.md +170 -237
- package/skills/using-aw-skills/hooks/session-start.sh +11 -41
- package/skills/zoom-out/SKILL.md +24 -0
- package/.cursor/rules/common-agents.md +0 -53
- package/.cursor/rules/common-aw-routing.md +0 -43
- package/.cursor/rules/common-coding-style.md +0 -52
- package/.cursor/rules/common-development-workflow.md +0 -33
- package/.cursor/rules/common-git-workflow.md +0 -28
- package/.cursor/rules/common-hooks.md +0 -34
- package/.cursor/rules/common-patterns.md +0 -35
- package/.cursor/rules/common-performance.md +0 -59
- package/.cursor/rules/common-security.md +0 -33
- package/.cursor/rules/common-testing.md +0 -33
- package/.cursor/skills/api-and-interface-design/SKILL.md +0 -75
- package/.cursor/skills/article-writing/SKILL.md +0 -85
- package/.cursor/skills/aw-brainstorm/SKILL.md +0 -115
- package/.cursor/skills/aw-build/SKILL.md +0 -152
- package/.cursor/skills/aw-build/evals/build-stage-cases.json +0 -28
- package/.cursor/skills/aw-debug/SKILL.md +0 -49
- package/.cursor/skills/aw-deploy/SKILL.md +0 -101
- package/.cursor/skills/aw-deploy/evals/deploy-stage-cases.json +0 -32
- package/.cursor/skills/aw-execute/SKILL.md +0 -47
- package/.cursor/skills/aw-execute/references/mode-code.md +0 -47
- package/.cursor/skills/aw-execute/references/mode-docs.md +0 -28
- package/.cursor/skills/aw-execute/references/mode-infra.md +0 -44
- package/.cursor/skills/aw-execute/references/mode-migration.md +0 -58
- package/.cursor/skills/aw-execute/references/worker-implementer.md +0 -26
- package/.cursor/skills/aw-execute/references/worker-parallel-worker.md +0 -23
- package/.cursor/skills/aw-execute/references/worker-quality-reviewer.md +0 -23
- package/.cursor/skills/aw-execute/references/worker-spec-reviewer.md +0 -23
- package/.cursor/skills/aw-execute/scripts/build-worker-bundle.js +0 -229
- package/.cursor/skills/aw-finish/SKILL.md +0 -111
- package/.cursor/skills/aw-investigate/SKILL.md +0 -109
- package/.cursor/skills/aw-plan/SKILL.md +0 -368
- package/.cursor/skills/aw-prepare/SKILL.md +0 -118
- package/.cursor/skills/aw-review/SKILL.md +0 -118
- package/.cursor/skills/aw-ship/SKILL.md +0 -115
- package/.cursor/skills/aw-spec/SKILL.md +0 -104
- package/.cursor/skills/aw-tasks/SKILL.md +0 -138
- package/.cursor/skills/aw-test/SKILL.md +0 -118
- package/.cursor/skills/aw-verify/SKILL.md +0 -51
- package/.cursor/skills/aw-yolo/SKILL.md +0 -111
- package/.cursor/skills/browser-testing-with-devtools/SKILL.md +0 -81
- package/.cursor/skills/bun-runtime/SKILL.md +0 -84
- package/.cursor/skills/ci-cd-and-automation/SKILL.md +0 -71
- package/.cursor/skills/code-simplification/SKILL.md +0 -74
- package/.cursor/skills/content-engine/SKILL.md +0 -88
- package/.cursor/skills/context-engineering/SKILL.md +0 -74
- package/.cursor/skills/deprecation-and-migration/SKILL.md +0 -75
- package/.cursor/skills/documentation-and-adrs/SKILL.md +0 -75
- package/.cursor/skills/documentation-lookup/SKILL.md +0 -90
- package/.cursor/skills/frontend-slides/SKILL.md +0 -184
- package/.cursor/skills/frontend-slides/STYLE_PRESETS.md +0 -330
- package/.cursor/skills/frontend-ui-engineering/SKILL.md +0 -68
- package/.cursor/skills/git-workflow-and-versioning/SKILL.md +0 -75
- package/.cursor/skills/idea-refine/SKILL.md +0 -84
- package/.cursor/skills/incremental-implementation/SKILL.md +0 -75
- package/.cursor/skills/investor-materials/SKILL.md +0 -96
- package/.cursor/skills/investor-outreach/SKILL.md +0 -76
- package/.cursor/skills/market-research/SKILL.md +0 -75
- package/.cursor/skills/mcp-server-patterns/SKILL.md +0 -67
- package/.cursor/skills/nextjs-turbopack/SKILL.md +0 -44
- package/.cursor/skills/performance-optimization/SKILL.md +0 -77
- package/.cursor/skills/security-and-hardening/SKILL.md +0 -70
- package/.cursor/skills/using-aw-skills/SKILL.md +0 -290
- package/.cursor/skills/using-aw-skills/evals/skill-trigger-cases.tsv +0 -25
- package/.cursor/skills/using-aw-skills/evals/test-skill-triggers.sh +0 -171
- package/.cursor/skills/using-aw-skills/hooks/hooks.json +0 -9
- package/.cursor/skills/using-aw-skills/hooks/session-start.sh +0 -67
- package/.cursor/skills/using-platform-skills/SKILL.md +0 -163
- package/.cursor/skills/using-platform-skills/evals/platform-selection-cases.json +0 -52
- /package/.cursor/rules/{golang-coding-style.md → golang-coding-style.mdc} +0 -0
- /package/.cursor/rules/{golang-hooks.md → golang-hooks.mdc} +0 -0
- /package/.cursor/rules/{golang-patterns.md → golang-patterns.mdc} +0 -0
- /package/.cursor/rules/{golang-security.md → golang-security.mdc} +0 -0
- /package/.cursor/rules/{golang-testing.md → golang-testing.mdc} +0 -0
- /package/.cursor/rules/{kotlin-coding-style.md → kotlin-coding-style.mdc} +0 -0
- /package/.cursor/rules/{kotlin-hooks.md → kotlin-hooks.mdc} +0 -0
- /package/.cursor/rules/{kotlin-patterns.md → kotlin-patterns.mdc} +0 -0
- /package/.cursor/rules/{kotlin-security.md → kotlin-security.mdc} +0 -0
- /package/.cursor/rules/{kotlin-testing.md → kotlin-testing.mdc} +0 -0
- /package/.cursor/rules/{php-coding-style.md → php-coding-style.mdc} +0 -0
- /package/.cursor/rules/{php-hooks.md → php-hooks.mdc} +0 -0
- /package/.cursor/rules/{php-patterns.md → php-patterns.mdc} +0 -0
- /package/.cursor/rules/{php-security.md → php-security.mdc} +0 -0
- /package/.cursor/rules/{php-testing.md → php-testing.mdc} +0 -0
- /package/.cursor/rules/{python-coding-style.md → python-coding-style.mdc} +0 -0
- /package/.cursor/rules/{python-hooks.md → python-hooks.mdc} +0 -0
- /package/.cursor/rules/{python-patterns.md → python-patterns.mdc} +0 -0
- /package/.cursor/rules/{python-security.md → python-security.mdc} +0 -0
- /package/.cursor/rules/{python-testing.md → python-testing.mdc} +0 -0
- /package/.cursor/rules/{swift-coding-style.md → swift-coding-style.mdc} +0 -0
- /package/.cursor/rules/{swift-hooks.md → swift-hooks.mdc} +0 -0
- /package/.cursor/rules/{swift-patterns.md → swift-patterns.mdc} +0 -0
- /package/.cursor/rules/{swift-security.md → swift-security.mdc} +0 -0
- /package/.cursor/rules/{swift-testing.md → swift-testing.mdc} +0 -0
- /package/.cursor/rules/{typescript-coding-style.md → typescript-coding-style.mdc} +0 -0
- /package/.cursor/rules/{typescript-hooks.md → typescript-hooks.mdc} +0 -0
- /package/.cursor/rules/{typescript-patterns.md → typescript-patterns.mdc} +0 -0
- /package/.cursor/rules/{typescript-security.md → typescript-security.mdc} +0 -0
- /package/.cursor/rules/{typescript-testing.md → typescript-testing.mdc} +0 -0
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
# Registry Structure & Path Resolution
|
|
2
|
+
|
|
3
|
+
How CASRE artifacts are organized in the AW registry and rules system.
|
|
4
|
+
|
|
5
|
+
## Two Namespace Models
|
|
6
|
+
|
|
7
|
+
### Platform namespace (shared, read-only)
|
|
8
|
+
|
|
9
|
+
Platform is flat — no team/sub_team level. Domains are the primary organization:
|
|
10
|
+
|
|
11
|
+
```
|
|
12
|
+
.aw/ # ← root anchor — all registry paths start here
|
|
13
|
+
.aw_registry/
|
|
14
|
+
platform/
|
|
15
|
+
<domain>/ # core, frontend, infra, data, sdet, review, design, services
|
|
16
|
+
skills/<slug>/SKILL.md
|
|
17
|
+
agents/<slug>.md
|
|
18
|
+
commands/<slug>.md
|
|
19
|
+
evals/<type>/<slug>/eval-*.md # colocated evals
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
**Domains:** core, frontend, infra, data, sdet, review, design, services
|
|
23
|
+
|
|
24
|
+
### Team namespaces
|
|
25
|
+
|
|
26
|
+
Teams always have `<team>/<sub_team>` and optionally nest further by domain:
|
|
27
|
+
|
|
28
|
+
```
|
|
29
|
+
.aw/
|
|
30
|
+
.aw_registry/
|
|
31
|
+
<team>/ # crm, leadgen, revex, mobile
|
|
32
|
+
<sub_team>/ # users, events, forms, courses, memberships, core
|
|
33
|
+
[<domain>/] # OPTIONAL: backend, core, frontend, infra, design, product, quality
|
|
34
|
+
skills/<slug>/SKILL.md
|
|
35
|
+
agents/<slug>.md
|
|
36
|
+
commands/<slug>.md
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
**Key differences:**
|
|
40
|
+
- **`platform`** uses `platform/<domain>/CAS` — no team layer
|
|
41
|
+
- **Teams** use `<team>/<sub_team>/CAS` or `<team>/<sub_team>/<domain>/CAS`
|
|
42
|
+
- Some teams put CAS directly: `crm/users/agents/` (no domain nesting)
|
|
43
|
+
- Others nest fully: `leadgen/events/backend/agents/` (with domain)
|
|
44
|
+
|
|
45
|
+
### Rules structure (separate hierarchy)
|
|
46
|
+
|
|
47
|
+
Rules live outside the registry in `.aw/.aw_rules/`:
|
|
48
|
+
|
|
49
|
+
```
|
|
50
|
+
.aw/
|
|
51
|
+
.aw_rules/ # Platform-level (shared)
|
|
52
|
+
platform/
|
|
53
|
+
<domain>/ # universal, security, frontend, backend, data, infra, sdet, mobile
|
|
54
|
+
AGENTS.md # Main rules file
|
|
55
|
+
references/<slug>.md # Individual rule references
|
|
56
|
+
evals/<slug>/eval-*.md # Colocated rule evals
|
|
57
|
+
<stack>/ # Optional stack overlays
|
|
58
|
+
AGENTS.md
|
|
59
|
+
references/<slug>.md
|
|
60
|
+
rule-manifest.json # Registry of all platform rules
|
|
61
|
+
|
|
62
|
+
.aw_rules.local/ # Team/repo-level (planned)
|
|
63
|
+
rule-manifest-local.json
|
|
64
|
+
render.json # Target mapping between platform + local layers
|
|
65
|
+
<path>/AGENTS.md # e.g., apps/billing/AGENTS.md
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
**Stack overlays:** `backend/nestjs/`, `backend/go-connect/`, `frontend/vue/`, `frontend/nuxt/`
|
|
69
|
+
|
|
70
|
+
## Path Resolution Flow
|
|
71
|
+
|
|
72
|
+
Use this decision tree to **construct** the exact target path. The rules are deterministic — walk the tree, substitute your variables, and you'll have the path. Searching is unnecessary because every combination of namespace + domain + type produces exactly one path.
|
|
73
|
+
|
|
74
|
+
```
|
|
75
|
+
1. Is this a RULE?
|
|
76
|
+
├── YES → Platform rule?
|
|
77
|
+
│ ├── YES → .aw/.aw_rules/platform/<domain>/references/<slug>.md
|
|
78
|
+
│ └── NO → .aw/.aw_rules.local/<path>/AGENTS.md
|
|
79
|
+
└── NO → continue
|
|
80
|
+
|
|
81
|
+
2. Is this PLATFORM work?
|
|
82
|
+
├── YES → Ask: which domain?
|
|
83
|
+
│ (core, frontend, infra, data, sdet, review, design, services)
|
|
84
|
+
│ → .aw/.aw_registry/platform/<domain>/<type>/<slug>
|
|
85
|
+
└── NO → continue
|
|
86
|
+
|
|
87
|
+
3. Is this TEAM work?
|
|
88
|
+
→ Ask: which team? (crm, leadgen, revex, mobile)
|
|
89
|
+
→ Ask: which sub_team? (users, events, courses, memberships, core, etc.)
|
|
90
|
+
→ Ask: need domain nesting? (optional)
|
|
91
|
+
├── NO domain → .aw/.aw_registry/<team>/<sub_team>/<type>/<slug>
|
|
92
|
+
└── YES domain → .aw/.aw_registry/<team>/<sub_team>/<domain>/<type>/<slug>
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
## Naming Conventions
|
|
96
|
+
|
|
97
|
+
`aw link` creates symlinks across all IDE directories (`.claude/`, `.cursor/`, `.codex/`) using **all-hyphens** names. The formula comes from `link.mjs`: it takes every directory name between the namespace root and the artifact type directory, appends the artifact slug, and joins them all with hyphens. When a team uses optional domain nesting (e.g., `revex/reselling/backend/agents/`), the domain becomes an extra segment in the name.
|
|
98
|
+
|
|
99
|
+
| Type | Platform | Team (flat) | Team (with domain) | Stage/core |
|
|
100
|
+
|---|---|---|---|---|
|
|
101
|
+
| Command | `aw:platform-core-plan` | `aw:revex-reselling-<slug>` | `aw:revex-reselling-backend-<slug>` | `aw:build` |
|
|
102
|
+
| Agent | `platform-data-db-engineer` | `revex-reselling-redis-reviewer` | `revex-reselling-backend-<slug>` | `code-reviewer` |
|
|
103
|
+
| Skill | `platform-core-architecture-design` | `revex-reselling-redis-patterns` | `revex-reselling-backend-<slug>` | `skill-creator` |
|
|
104
|
+
| Rule | `<domain>/<slug>` | repo-local path | repo-local path | — |
|
|
105
|
+
|
|
106
|
+
Platform and team (flat) examples are real names from the live system. When in doubt, verify against what `aw link` actually created — the symlinks are the source of truth:
|
|
107
|
+
- Skills: `ls ~/.claude/skills/` (or `.cursor/`, `.codex/`)
|
|
108
|
+
- Agents: `ls ~/.claude/agents/`
|
|
109
|
+
- Commands: `ls ~/.claude/commands/aw/`
|
|
110
|
+
|
|
111
|
+
**Examples of cross-artifact references (all use the hyphen-joined symlink name):**
|
|
112
|
+
|
|
113
|
+
```yaml
|
|
114
|
+
# Agent loading skills
|
|
115
|
+
skills:
|
|
116
|
+
- revex-reselling-redis-patterns
|
|
117
|
+
- platform-data-mongodb-patterns
|
|
118
|
+
|
|
119
|
+
# Command loading agents
|
|
120
|
+
agents:
|
|
121
|
+
- platform-review-security-reviewer
|
|
122
|
+
- platform-review-performance-reviewer
|
|
123
|
+
|
|
124
|
+
# subagent_type in commands/skills
|
|
125
|
+
subagent_type: "platform-review-security-reviewer"
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
These all match the symlink names in `~/.claude/skills/` and `~/.claude/agents/` that `aw link` creates.
|
|
129
|
+
|
|
130
|
+
## Colocated Eval Placement
|
|
131
|
+
|
|
132
|
+
Evals live next to their parent artifact, not in a top-level `evals/` directory:
|
|
133
|
+
|
|
134
|
+
| Parent Type | Eval Location |
|
|
135
|
+
|---|---|
|
|
136
|
+
| Skill | `skills/<slug>/evals/eval-*.md` (inside skill dir) |
|
|
137
|
+
| Agent | `agents/evals/<slug>/eval-*.md` (sibling evals/ dir) |
|
|
138
|
+
| Command | `commands/evals/<slug>/eval-*.md` (sibling evals/ dir) |
|
|
139
|
+
| Rule | `rules/evals/<slug>/eval-*.md` or within `.aw/.aw_rules/` references |
|
|
140
|
+
| Eval (meta) | `evals/evals/eval-*.md` (self-referential) |
|
|
141
|
+
|
|
142
|
+
## How `aw pull` Maps to IDE Paths
|
|
143
|
+
|
|
144
|
+
After `aw pull`, registry artifacts are synced to IDE-local locations:
|
|
145
|
+
|
|
146
|
+
| Registry | Claude Code | Cursor | Codex |
|
|
147
|
+
|---|---|---|---|
|
|
148
|
+
| `skills/<slug>/SKILL.md` | `.claude/skills/<slug>/SKILL.md` | `.cursor/rules/<slug>.mdc` | `.codex/<slug>/` |
|
|
149
|
+
| `agents/<slug>.md` | `.claude/agents/<slug>.md` | `.cursor/rules/<slug>.mdc` | `.codex/<slug>/` |
|
|
150
|
+
| `commands/<slug>.md` | `.claude/commands/<slug>.md` | N/A (manual) | N/A |
|
|
151
|
+
|
|
152
|
+
`skills-lock.json` tracks installed skills with SHA256 integrity hashes (like package-lock.json for skills).
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Rubric: Agent Quality (10 dimensions, /100)
|
|
2
|
+
|
|
3
|
+
Evaluates the quality of an AW agent definition file.
|
|
4
|
+
|
|
5
|
+
## Scoring Table
|
|
6
|
+
|
|
7
|
+
| # | Dimension | 0 (Missing) | 5 (Partial) | 10 (Complete) |
|
|
8
|
+
|---|-----------|-------------|-------------|---------------|
|
|
9
|
+
| 1 | **Identity & Personality** | No identity defined | Has role but missing personality, memory, or experience | Full 4-field identity: role, personality, memory, experience |
|
|
10
|
+
| 2 | **Core Mission** | Missing or vague | 1 generic sentence | 2-3 specific sentences naming domain, outcomes, scope boundaries |
|
|
11
|
+
| 3 | **Critical Rules** | Missing or soft suggestions | 1-2 rules without BLOCK/NEVER keywords | 3-5 hard rules with BLOCK/NEVER/ALWAYS and measurable thresholds |
|
|
12
|
+
| 4 | **Process / Workflow** | Missing | Numbered list without code or commands | Step-by-step with input/output per step, code examples, bash commands |
|
|
13
|
+
| 5 | **Deliverables** | Missing | Table with names only | Table with format + quality bar + inline template for each deliverable |
|
|
14
|
+
| 6 | **Communication Style** | Missing or 1 sentence | 2 personality traits described | 3-4 example phrases showing distinct voice and tone |
|
|
15
|
+
| 7 | **Code Examples** | No code blocks | 1 generic block | 2+ domain-specific blocks showing good vs. bad patterns |
|
|
16
|
+
| 8 | **Learning & Memory** | Missing | Lists what to remember | Pattern recognition + anti-patterns + cross-session learning protocol |
|
|
17
|
+
| 9 | **Success Metrics** | Missing or vague | 2-3 metrics without numbers | 4-5 quantified targets with explicit thresholds |
|
|
18
|
+
| 10 | **Advanced Capabilities** | Missing | 1-2 bullets | 3+ mastery areas showing growth trajectory from basic to expert |
|
|
19
|
+
|
|
20
|
+
## Tier Thresholds
|
|
21
|
+
|
|
22
|
+
| Tier | Score | Interpretation |
|
|
23
|
+
|------|-------|----------------|
|
|
24
|
+
| **S** | 90-100 | Production-grade. Agent is distinctive, reliable, and self-improving. |
|
|
25
|
+
| **A** | 75-89 | Strong. Minor gaps in voice, metrics, or advanced capabilities. |
|
|
26
|
+
| **B** | 60-74 | Functional. Usable but lacks personality depth or measurable targets. |
|
|
27
|
+
| **C** | 40-59 | Draft. Core mission exists but workflow and deliverables need work. |
|
|
28
|
+
| **D** | 0-39 | Stub or broken. Not usable without significant rewrite. |
|
|
29
|
+
|
|
30
|
+
## How to use this rubric
|
|
31
|
+
|
|
32
|
+
1. Open the agent definition file.
|
|
33
|
+
2. Score each dimension independently (0, 5, or 10).
|
|
34
|
+
3. Sum all 10 scores for a total out of 100.
|
|
35
|
+
4. Map the total to a tier using the thresholds above.
|
|
36
|
+
5. Prioritize fixing dimensions that scored 0 -- these represent missing sections that block agent effectiveness.
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Rubric: Command Quality (10 dimensions, /100)
|
|
2
|
+
|
|
3
|
+
Evaluates the quality of an AW command definition file.
|
|
4
|
+
|
|
5
|
+
## Scoring Table
|
|
6
|
+
|
|
7
|
+
| # | Dimension | 0 (Missing) | 5 (Partial) | 10 (Complete) |
|
|
8
|
+
|---|-----------|-------------|-------------|---------------|
|
|
9
|
+
| 1 | **Frontmatter** | Missing | `name` + `description` present | `name` + `description` + `argument-hint` + `mcp` list |
|
|
10
|
+
| 2 | **Protocol Reference** | Missing | Generic mention of a protocol | Explicit AW-PROTOCOL reference + skill loading gate requirement |
|
|
11
|
+
| 3 | **Agent Roster** | Missing | Informal mentions of agents in prose | Table with phase, agent name, tier, and skills loaded per agent |
|
|
12
|
+
| 4 | **Skill Loading Gate** | Missing | Partial mention of skill loading | Full blocking gate: agent -> skills -> learnings chain before execution |
|
|
13
|
+
| 5 | **Phase Structure** | No phases defined | Phases listed without I/O contracts | Numbered phases with input, output, and human checkpoints per phase |
|
|
14
|
+
| 6 | **Output Format** | Missing | Vague description | Concrete markdown template with placeholders and field names |
|
|
15
|
+
| 7 | **Human Checkpoints** | None defined | 1 vague mention of review | Explicit gates with approval criteria and proceed/halt behavior |
|
|
16
|
+
| 8 | **Learning / Transparency** | Missing | Mentions learning broadly | Per-step .md artifacts + transparency JSON + learnings append protocol |
|
|
17
|
+
| 9 | **User Communication** | Silent execution | Some status messages | Status message per phase with progress indicator format |
|
|
18
|
+
| 10 | **Error Handling** | Missing | "Retry if failed" | Per-phase failure paths, fallback agents, and halt conditions |
|
|
19
|
+
|
|
20
|
+
## Tier Thresholds
|
|
21
|
+
|
|
22
|
+
| Tier | Score | Interpretation |
|
|
23
|
+
|------|-------|----------------|
|
|
24
|
+
| **S** | 90-100 | Production-grade. Fully orchestrated with checkpoints and error recovery. |
|
|
25
|
+
| **A** | 75-89 | Strong. Minor gaps in error handling or transparency artifacts. |
|
|
26
|
+
| **B** | 60-74 | Functional. Phases exist but missing checkpoints or fallback paths. |
|
|
27
|
+
| **C** | 40-59 | Draft. Basic structure present but no orchestration rigor. |
|
|
28
|
+
| **D** | 0-39 | Stub or broken. Not executable as a command. |
|
|
29
|
+
|
|
30
|
+
## How to use this rubric
|
|
31
|
+
|
|
32
|
+
1. Open the command definition file.
|
|
33
|
+
2. Score each dimension independently (0, 5, or 10).
|
|
34
|
+
3. Sum all 10 scores for a total out of 100.
|
|
35
|
+
4. Map the total to a tier using the thresholds above.
|
|
36
|
+
5. Commands scoring below B-tier should not be shipped to users -- focus on phase structure and error handling first.
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Rubric: Eval Quality (10 dimensions, /100)
|
|
2
|
+
|
|
3
|
+
Evaluates the quality of an AW eval definition file used for benchmarking agent and skill performance.
|
|
4
|
+
|
|
5
|
+
## Scoring Table
|
|
6
|
+
|
|
7
|
+
| # | Dimension | 0 (Missing) | 5 (Partial) | 10 (Complete) |
|
|
8
|
+
|---|-----------|-------------|-------------|---------------|
|
|
9
|
+
| 1 | **Frontmatter** | Missing | `name` + `target` present | `name` + `target` + `category` + `difficulty` |
|
|
10
|
+
| 2 | **Task Clarity** | Missing or vague | Describes intent in general terms | Concrete task with specific artifact type, namespace, and domain |
|
|
11
|
+
| 3 | **Context** | Missing | Minimal (just a path or name) | Namespace, domain, target path, existing related work, key packages |
|
|
12
|
+
| 4 | **Expected Outcomes** | Missing | 1-2 vague expectations | 4+ specific, verifiable checkboxes with concrete acceptance criteria |
|
|
13
|
+
| 5 | **Grading Criteria** | Missing | Just PASS/FAIL | PASS/PARTIAL/FAIL with clear thresholds for each grade |
|
|
14
|
+
| 6 | **Evaluation Method** | Missing | Unspecified or implied | Explicit: deterministic, model-based, or hybrid with rationale |
|
|
15
|
+
| 7 | **Scenario Diversity** | Single happy path only | Happy path + 1 edge case | Happy + failure + edge case + adversarial scenario |
|
|
16
|
+
| 8 | **False-Pass Resistance** | Would pass a clearly wrong artifact | Some specificity in assertions | Assertions target quality dimensions, not just existence checks |
|
|
17
|
+
| 9 | **Reproducibility** | Non-deterministic or vague setup | Mostly reproducible with minor variance | Fully deterministic or clearly bounded stochastic with seed/tolerance |
|
|
18
|
+
| 10 | **Baseline Tracking** | No baseline mentioned | Mentions baseline informally | Explicit with/without comparison methodology and stored baseline scores |
|
|
19
|
+
|
|
20
|
+
## Tier Thresholds
|
|
21
|
+
|
|
22
|
+
| Tier | Score | Interpretation |
|
|
23
|
+
|------|-------|----------------|
|
|
24
|
+
| **S** | 90-100 | Production-grade. Reliable, reproducible, and resistant to false passes. |
|
|
25
|
+
| **A** | 75-89 | Strong. Good coverage; minor gaps in diversity or baseline tracking. |
|
|
26
|
+
| **B** | 60-74 | Functional. Tests the right thing but may miss edge cases or allow false passes. |
|
|
27
|
+
| **C** | 40-59 | Draft. Task is defined but grading criteria or scenarios need work. |
|
|
28
|
+
| **D** | 0-39 | Stub or broken. Not usable for benchmarking. |
|
|
29
|
+
|
|
30
|
+
## How to use this rubric
|
|
31
|
+
|
|
32
|
+
1. Open the eval definition file.
|
|
33
|
+
2. Score each dimension independently (0, 5, or 10).
|
|
34
|
+
3. Sum all 10 scores for a total out of 100.
|
|
35
|
+
4. Map the total to a tier using the thresholds above.
|
|
36
|
+
5. Evals below B-tier should not be included in benchmark suites -- prioritize false-pass resistance and scenario diversity to avoid misleading results.
|
|
@@ -0,0 +1,132 @@
|
|
|
1
|
+
# Rubric: Meta-Evaluation
|
|
2
|
+
|
|
3
|
+
Score an eval's quality across 5 dimensions. Total: /50.
|
|
4
|
+
|
|
5
|
+
## Dimensions
|
|
6
|
+
|
|
7
|
+
### 1. Scenario Diversity (0-10)
|
|
8
|
+
|
|
9
|
+
How broadly does the eval cover the artifact's behavior space?
|
|
10
|
+
|
|
11
|
+
| Score | Description |
|
|
12
|
+
|-------|-------------|
|
|
13
|
+
| 0 | Single happy-path scenario only |
|
|
14
|
+
| 2 | Happy path with minor input variations |
|
|
15
|
+
| 5 | Happy path + at least one edge case or failure scenario |
|
|
16
|
+
| 7 | Happy path + failure + edge case coverage |
|
|
17
|
+
| 10 | Happy path + failure + edge case + adversarial/ambiguous inputs |
|
|
18
|
+
|
|
19
|
+
**What to look for:**
|
|
20
|
+
- Does the eval test what happens when required fields are missing?
|
|
21
|
+
- Are boundary conditions exercised (empty input, maximum length, special characters)?
|
|
22
|
+
- Is there at least one scenario where the artifact should refuse or fail gracefully?
|
|
23
|
+
- Are adversarial prompts included (conflicting instructions, prompt injection attempts)?
|
|
24
|
+
|
|
25
|
+
### 2. Grader Determinism (0-10)
|
|
26
|
+
|
|
27
|
+
How reproducible are the eval results across runs?
|
|
28
|
+
|
|
29
|
+
| Score | Description |
|
|
30
|
+
|-------|-------------|
|
|
31
|
+
| 0 | Fully subjective -- human judgment with no rubric |
|
|
32
|
+
| 3 | Model-based grader with vague instructions ("is this good?") |
|
|
33
|
+
| 5 | Model-based grader with specific criteria and examples |
|
|
34
|
+
| 7 | Mix of deterministic checks and bounded model grading |
|
|
35
|
+
| 10 | Fully deterministic script or model grader with explicit pass/fail boundaries |
|
|
36
|
+
|
|
37
|
+
**What to look for:**
|
|
38
|
+
- Are pass/fail criteria unambiguous enough that two reviewers would agree?
|
|
39
|
+
- Does the grader use exact-match, regex, or structured checks where possible?
|
|
40
|
+
- If model-based, does the grader prompt include concrete examples of pass and fail?
|
|
41
|
+
- Could you automate this grader in CI without human intervention?
|
|
42
|
+
|
|
43
|
+
### 3. False-Pass Resistance (0-10)
|
|
44
|
+
|
|
45
|
+
Would a clearly wrong artifact still pass the eval?
|
|
46
|
+
|
|
47
|
+
| Score | Description |
|
|
48
|
+
|-------|-------------|
|
|
49
|
+
| 0 | A boilerplate or empty artifact would pass |
|
|
50
|
+
| 3 | Checks for presence of output but not correctness |
|
|
51
|
+
| 5 | Some specificity -- checks named sections or keywords |
|
|
52
|
+
| 7 | Targeted assertions on content, structure, and relationships |
|
|
53
|
+
| 10 | Assertions that demonstrably fail on known-wrong artifacts |
|
|
54
|
+
|
|
55
|
+
**What to look for:**
|
|
56
|
+
- Does the eval check content meaning, not just content existence?
|
|
57
|
+
- Are there negative assertions (artifact must NOT contain X)?
|
|
58
|
+
- Would a copy-paste of the prompt back as output pass? If yes, score low.
|
|
59
|
+
- Does the eval verify relationships between parts (e.g., summary matches detail)?
|
|
60
|
+
|
|
61
|
+
### 4. Criteria Specificity (0-10)
|
|
62
|
+
|
|
63
|
+
How precisely are success criteria defined?
|
|
64
|
+
|
|
65
|
+
| Score | Description |
|
|
66
|
+
|-------|-------------|
|
|
67
|
+
| 0 | Vague ("output should be good", "looks correct") |
|
|
68
|
+
| 3 | Named qualities without measurement ("should be comprehensive") |
|
|
69
|
+
| 5 | Named sections or fields to check with qualitative descriptions |
|
|
70
|
+
| 7 | Quantified thresholds for most criteria (counts, percentages, sizes) |
|
|
71
|
+
| 10 | All criteria have quantified thresholds with evidence requirements |
|
|
72
|
+
|
|
73
|
+
**What to look for:**
|
|
74
|
+
- Can you turn each criterion into a binary pass/fail without interpretation?
|
|
75
|
+
- Are numeric thresholds stated (e.g., "at least 3 scenarios", "under 200 words")?
|
|
76
|
+
- Does the eval require evidence citations or examples in the output?
|
|
77
|
+
- Are edge cases in the criteria themselves addressed (what counts as "partial")?
|
|
78
|
+
|
|
79
|
+
### 5. Baseline Tracking (0-10)
|
|
80
|
+
|
|
81
|
+
Does the eval support measuring improvement over time?
|
|
82
|
+
|
|
83
|
+
| Score | Description |
|
|
84
|
+
|-------|-------------|
|
|
85
|
+
| 0 | No baseline, no comparison methodology |
|
|
86
|
+
| 3 | Mentions that results should be compared but no method |
|
|
87
|
+
| 5 | References a baseline or before/after comparison |
|
|
88
|
+
| 7 | Explicit with/without methodology with defined metrics |
|
|
89
|
+
| 10 | Reproducible baseline with versioned artifacts, stored results, and diff method |
|
|
90
|
+
|
|
91
|
+
**What to look for:**
|
|
92
|
+
- Is there a "before" snapshot or known-good baseline to compare against?
|
|
93
|
+
- Can you re-run the eval on an older version and get comparable results?
|
|
94
|
+
- Does the eval track scores over time (not just pass/fail)?
|
|
95
|
+
- Is the comparison methodology documented enough for someone else to reproduce?
|
|
96
|
+
|
|
97
|
+
## Tier Thresholds
|
|
98
|
+
|
|
99
|
+
| Tier | Score | Interpretation |
|
|
100
|
+
|------|-------|----------------|
|
|
101
|
+
| **S** | 45-50 | Production-grade eval. Ship it. |
|
|
102
|
+
| **A** | 38-44 | Strong eval. Minor gaps acceptable for non-critical artifacts. |
|
|
103
|
+
| **B** | 30-37 | Adequate eval. Acceptable for first iteration; improve before relying on it for regressions. |
|
|
104
|
+
| **C** | 20-29 | Weak eval. Provides some signal but has significant blind spots. |
|
|
105
|
+
| **D** | 0-19 | Eval provides negligible quality signal. Rewrite before using. |
|
|
106
|
+
|
|
107
|
+
## How to Use
|
|
108
|
+
|
|
109
|
+
1. **Score each eval after writing it.** Fill in the 5 dimensions honestly. If you authored the eval, have someone else score it -- author bias inflates scores by 5-10 points on average.
|
|
110
|
+
|
|
111
|
+
2. **Gate on tier before merging.** New evals for skills and agents should be B-tier or above. Critical-path artifacts (commands, platform rules) should target A-tier.
|
|
112
|
+
|
|
113
|
+
3. **Use dimension scores to guide improvements.** A low Scenario Diversity score is fixed by adding scenarios. A low Grader Determinism score requires rewriting the grader. Address the lowest-scoring dimension first for maximum impact.
|
|
114
|
+
|
|
115
|
+
4. **Re-score after changes.** When you modify an eval, re-run the rubric. Track the score in the eval's frontmatter or a changelog comment.
|
|
116
|
+
|
|
117
|
+
5. **Compare across evals.** Use tier distribution to gauge overall eval maturity for a project. If most evals are C-tier, invest in eval quality before adding more artifacts.
|
|
118
|
+
|
|
119
|
+
### Scoring Template
|
|
120
|
+
|
|
121
|
+
```markdown
|
|
122
|
+
## Meta-Eval Score
|
|
123
|
+
|
|
124
|
+
| Dimension | Score | Notes |
|
|
125
|
+
|-----------|-------|-------|
|
|
126
|
+
| Scenario Diversity | /10 | |
|
|
127
|
+
| Grader Determinism | /10 | |
|
|
128
|
+
| False-Pass Resistance | /10 | |
|
|
129
|
+
| Criteria Specificity | /10 | |
|
|
130
|
+
| Baseline Tracking | /10 | |
|
|
131
|
+
| **Total** | **/50** | **Tier: ?** |
|
|
132
|
+
```
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Rubric: Rule Quality (10 dimensions, /100)
|
|
2
|
+
|
|
3
|
+
Evaluates the quality of an AW rule definition file. Expands the original 5-dimension /50 rubric from aw-rules to 10 dimensions /100.
|
|
4
|
+
|
|
5
|
+
## Scoring Table
|
|
6
|
+
|
|
7
|
+
| # | Dimension | 0 (Missing) | 5 (Partial) | 10 (Complete) |
|
|
8
|
+
|---|-----------|-------------|-------------|---------------|
|
|
9
|
+
| 1 | **Frontmatter** | Missing | `id` + `severity` present | `id` + `severity` + `domains` + `paths` glob patterns |
|
|
10
|
+
| 2 | **Rule Statement** | Missing | Vague or ambiguous phrasing | One clear sentence stating the requirement and why it matters |
|
|
11
|
+
| 3 | **WRONG Example** | Missing | Generic or hypothetical violation | Real violation pattern drawn from actual codebase conventions |
|
|
12
|
+
| 4 | **RIGHT Example** | Missing | Generic fix | Verified fix grounded in platform docs or referenced skills |
|
|
13
|
+
| 5 | **Skill Link** | Missing | Wrong or broken link | Correct link to an existing, relevant skill |
|
|
14
|
+
| 6 | **Severity Justification** | Missing | Just states MUST/SHOULD without rationale | Explains why this severity: risk, blast radius, precedent |
|
|
15
|
+
| 7 | **Automation Path** | Missing | "Can be linted" without specifics | Specific lint rule, pre-commit hook, or CI script that enforces this |
|
|
16
|
+
| 8 | **Scope Precision** | Too broad ("all code") | Domain-scoped (e.g., "backend" or "frontend") | Path-scoped with glob patterns (e.g., `services/*/src/**/*.ts`) |
|
|
17
|
+
| 9 | **Exceptions** | No mention of edge cases | "Edge cases exist" without detail | Explicit exceptions listed with justification for each |
|
|
18
|
+
| 10 | **Manifest Entry** | Missing from rule-manifest.json | Incomplete entry (missing fields) | Full entry: id, severity, domains, description, principle |
|
|
19
|
+
|
|
20
|
+
## Tier Thresholds
|
|
21
|
+
|
|
22
|
+
| Tier | Score | Interpretation |
|
|
23
|
+
|------|-------|----------------|
|
|
24
|
+
| **S** | 90-100 | Production-grade. Enforceable, scoped, and documented with automation. |
|
|
25
|
+
| **A** | 75-89 | Strong. Clear rule with examples; minor gaps in automation or exceptions. |
|
|
26
|
+
| **B** | 60-74 | Functional. Rule is understandable but lacks enforcement path or scope precision. |
|
|
27
|
+
| **C** | 40-59 | Draft. Statement exists but missing examples or justification. |
|
|
28
|
+
| **D** | 0-39 | Stub or broken. Not enforceable or understandable. |
|
|
29
|
+
|
|
30
|
+
## How to use this rubric
|
|
31
|
+
|
|
32
|
+
1. Open the rule file and its corresponding entry in `rule-manifest.json`.
|
|
33
|
+
2. Score each dimension independently (0, 5, or 10).
|
|
34
|
+
3. Sum all 10 scores for a total out of 100.
|
|
35
|
+
4. Map the total to a tier using the thresholds above.
|
|
36
|
+
5. Rules below B-tier should not be added to the active manifest -- prioritize adding WRONG/RIGHT examples and an automation path.
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Rubric: Skill Quality (10 dimensions, /100)
|
|
2
|
+
|
|
3
|
+
Evaluates the quality of an AW skill file (SKILL.md + references/).
|
|
4
|
+
|
|
5
|
+
## Scoring Table
|
|
6
|
+
|
|
7
|
+
| # | Dimension | 0 (Missing) | 5 (Partial) | 10 (Complete) |
|
|
8
|
+
|---|-----------|-------------|-------------|---------------|
|
|
9
|
+
| 1 | **Frontmatter** | Missing | `name` + `description` present | `name` + `description` + `trigger` with "use when" clause |
|
|
10
|
+
| 2 | **Purpose Statement** | Missing | 1 vague sentence | 2-3 specific sentences naming domain, scope, and outcomes |
|
|
11
|
+
| 3 | **When to Use** | Missing | 1 trigger scenario | 3+ trigger scenarios covering distinct use cases |
|
|
12
|
+
| 4 | **Instructions** | Missing or vague | Numbered list without detail | Step-by-step with concrete actions, commands, decision points |
|
|
13
|
+
| 5 | **Code Examples** | No code blocks | 1 generic code block | 3+ domain-specific code blocks with context |
|
|
14
|
+
| 6 | **Checklists** | Missing | Basic bullet list | Items with pass/fail criteria, severity, and fix guidance |
|
|
15
|
+
| 7 | **References** | No references | Internal links only | Links to platform-docs + reference files in references/ |
|
|
16
|
+
| 8 | **Progressive Disclosure** | Everything in SKILL.md or too sparse | SKILL.md + some refs | SKILL.md < 5k words, detailed content in references/ |
|
|
17
|
+
| 9 | **Domain Specificity** | Generic / framework-agnostic | Mentions package names | Actual package names, design tokens, API patterns with versions |
|
|
18
|
+
| 10 | **Output Format** | No output template | Vague description of output | Concrete markdown template with field names and placeholders |
|
|
19
|
+
|
|
20
|
+
## Tier Thresholds
|
|
21
|
+
|
|
22
|
+
| Tier | Score | Interpretation |
|
|
23
|
+
|------|-------|----------------|
|
|
24
|
+
| **S** | 90-100 | Production-grade. Ready for cross-team adoption. |
|
|
25
|
+
| **A** | 75-89 | Strong. Minor gaps in examples or references. |
|
|
26
|
+
| **B** | 60-74 | Functional. Usable but missing depth in 2-3 dimensions. |
|
|
27
|
+
| **C** | 40-59 | Draft. Needs significant work before team use. |
|
|
28
|
+
| **D** | 0-39 | Stub or broken. Not usable without rewrite. |
|
|
29
|
+
|
|
30
|
+
## How to use this rubric
|
|
31
|
+
|
|
32
|
+
1. Open the skill's `SKILL.md` and its `references/` directory.
|
|
33
|
+
2. Score each dimension independently (0, 5, or 10).
|
|
34
|
+
3. Sum all 10 scores for a total out of 100.
|
|
35
|
+
4. Map the total to a tier using the thresholds above.
|
|
36
|
+
5. Record dimension-level scores to guide targeted improvements -- focus on any dimension scoring 0 or 5 first.
|