@skill-graph/cli 0.5.7 → 0.5.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +27 -3
- package/README.md +40 -14
- package/SKILL_GRAPH.md +2 -2
- package/bin/skill-graph.js +118 -2
- package/docs/ADOPTION.md +1 -1
- package/docs/PRIMER.md +6 -5
- package/docs/QUICKSTART-30MIN.md +2 -2
- package/docs/SKILL_AUDIT_CHECKLIST.md +1 -1
- package/docs/SKILL_METADATA_PROTOCOL.md +2 -2
- package/docs/_archived/marketplace-publication-priority-2026-05-18.md +1 -1
- package/docs/_drafts/0.5.8-release-prep.md +164 -0
- package/docs/field-reference.generated.md +1 -1
- package/docs/field-reference.md +2 -2
- package/docs/manifest-field-mapping.md +3 -3
- package/docs/marketplace-publication-queue.generated.md +2 -2
- package/docs/plans/scripts-roadmap.md +2 -2
- package/docs/positioning.md +88 -0
- package/docs/research/skill-comprehension-eval-research.md +5 -5
- package/docs/research/skill-demand-gap-roadmap-2026-05-16.md +215 -0
- package/docs/status.generated.md +48 -0
- package/examples/audits/context-graph/findings.md +59 -0
- package/examples/audits/context-graph/scorecard.md +22 -0
- package/examples/audits/context-graph/verdict.md +33 -0
- package/examples/evals/a11y.json +45 -13
- package/examples/evals/api-design.json +18 -5
- package/examples/evals/code-review.json +18 -5
- package/examples/evals/data-modeling.json +18 -5
- package/examples/evals/database-migration.json +18 -5
- package/examples/evals/debugging.json +37 -11
- package/examples/evals/dependency-architecture.json +18 -5
- package/examples/evals/design-system-architecture.json +18 -5
- package/examples/evals/error-tracking.json +18 -5
- package/examples/evals/event-contract-design.json +18 -5
- package/examples/evals/form-ux-architecture.json +18 -5
- package/examples/evals/framework-fit-analysis.json +18 -5
- package/examples/evals/graph-audit.json +55 -13
- package/examples/evals/information-architecture.json +18 -5
- package/examples/evals/interaction-feedback.json +18 -5
- package/examples/evals/interaction-patterns.json +18 -5
- package/examples/evals/layout-composition.json +18 -5
- package/examples/evals/lint-overlay.json +38 -11
- package/examples/evals/microcopy.json +18 -5
- package/examples/evals/observability-modeling.json +18 -5
- package/examples/evals/pattern-recognition.json +32 -9
- package/examples/evals/performance-engineering.json +18 -5
- package/examples/evals/refactor.json +41 -12
- package/examples/evals/semiotics.json +18 -5
- package/examples/evals/skill-infrastructure.json +32 -9
- package/examples/evals/skill-router.json +42 -13
- package/examples/evals/system-interface-contracts.json +18 -5
- package/examples/evals/task-analysis.json +18 -5
- package/examples/evals/testing-strategy.json +36 -11
- package/examples/evals/type-safety.json +251 -66
- package/examples/evals/visual-design-foundations.json +18 -5
- package/examples/evals/webhook-integration.json +18 -5
- package/examples/fixture-skills/README.md +47 -0
- package/examples/fixture-skills/comprehension-full/SKILL.md +79 -0
- package/examples/fixture-skills/minimal-capability/SKILL.md +51 -0
- package/examples/fixture-skills/with-grounding/SKILL.md +78 -0
- package/examples/fixture-skills/with-relations/SKILL.md +87 -0
- package/examples/skills.manifest.sample.json +1722 -446
- package/marketplace/README.md +1 -1
- package/marketplace/skills/a11y/SKILL.md +1 -1
- package/marketplace/skills/best-practice/SKILL.md +211 -0
- package/marketplace/skills/context-graph/SKILL.md +1 -1
- package/marketplace/skills/debugging/SKILL.md +1 -1
- package/marketplace/skills/graph-audit/SKILL.md +3 -1
- package/marketplace/skills/postgres-rls/SKILL.md +284 -0
- package/marketplace/skills/refactor/SKILL.md +1 -1
- package/marketplace/skills/skill-infrastructure/SKILL.md +2 -0
- package/marketplace/skills/skill-router/SKILL.md +3 -1
- package/marketplace/skills/testing-strategy/SKILL.md +1 -1
- package/package.json +3 -1
- package/scripts/__tests__/test-marketplace-export.js +6 -2
- package/scripts/__tests__/test-v3-1-alias-contract.js +3 -3
- package/scripts/build-status-doc.js +177 -0
- package/scripts/check-doc-drift.js +224 -0
- package/scripts/check-markdown-links.js +34 -4
- package/scripts/check-mirror-freeze.js +270 -0
- package/scripts/export-marketplace-skills.js +35 -6
- package/scripts/lib/audit-prompt-builder.js +3 -3
- package/scripts/lib/parse-frontmatter.js +2 -2
- package/scripts/skill-audit.js +7 -9
|
@@ -0,0 +1,164 @@
|
|
|
1
|
+
# 0.5.8 Release Prep — Review Before Commit/Publish
|
|
2
|
+
|
|
3
|
+
> **Status:** DRAFT — not yet applied to `package.json` or `CHANGELOG.md`.
|
|
4
|
+
> **Prepared:** 2026-05-19 by Karpathy-loop Phase 2 Item 34.
|
|
5
|
+
> **Action required:** review the proposed CHANGELOG entry and version bump, then either apply directly via the commands at the bottom or hand-edit.
|
|
6
|
+
|
|
7
|
+
## Proposed version bump
|
|
8
|
+
|
|
9
|
+
`package.json`:
|
|
10
|
+
|
|
11
|
+
```diff
|
|
12
|
+
- "version": "0.5.7",
|
|
13
|
+
+ "version": "0.5.8",
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
## Proposed CHANGELOG entry
|
|
17
|
+
|
|
18
|
+
Insert as a new section between the existing `## [Unreleased]` header and the next `## [0.5.0]` section (the existing `[Unreleased]` block, currently containing pre-Phase-2 entries, should be **promoted** to the body of `## [0.5.8]`; a fresh empty `## [Unreleased]` header replaces it):
|
|
19
|
+
|
|
20
|
+
```markdown
|
|
21
|
+
## [Unreleased]
|
|
22
|
+
|
|
23
|
+
## [0.5.8] — 2026-05-19
|
|
24
|
+
|
|
25
|
+
The "Karpathy-loop Phase 2" release. Ships the deterministic-drift sentinels,
|
|
26
|
+
the mirror-freeze linter, the generated status doc, the `doctor` subcommand,
|
|
27
|
+
the first hermetic test fixture, and the positioning + onboarding rework that
|
|
28
|
+
Phase 2 specified. All Phase 1 audit gates remained green throughout; the
|
|
29
|
+
release also includes the 33 Phase 1 truth-repair commits.
|
|
30
|
+
|
|
31
|
+
### Added
|
|
32
|
+
|
|
33
|
+
- **`scripts/check-doc-drift.js`** — schema_version drift sentinel that
|
|
34
|
+
scans active docs for stale `schema_version: N` references and refuses
|
|
35
|
+
to teach an old contract as current. Reads the active version from
|
|
36
|
+
`schemas/skill.schema.json`; allowlists `_archived/`, `docs/migrations/`,
|
|
37
|
+
`examples/`, `CHANGELOG.md`, and `vN → vM` migration sections. Flags:
|
|
38
|
+
`--json`, `--quiet`, `--include-warn`.
|
|
39
|
+
- **`scripts/check-mirror-freeze.js`** — fails on active-source / active-
|
|
40
|
+
package claims in the two docs-only deprecation mirrors
|
|
41
|
+
(`skill-metadata-protocol`, `skill-audit-loop`). Detects `@skill-graph/
|
|
42
|
+
protocol`, `@skill-graph/audit`, `src/`, `lib/schemas/`, `schemas/skill.v*.
|
|
43
|
+
schema.json`, and `evals/` references that contradict the post-ADR-0009
|
|
44
|
+
docs-only status. File-level banner detection skips files explicitly
|
|
45
|
+
stamped as historical / frozen / deprecation-mirror snapshots.
|
|
46
|
+
- **`scripts/build-status-doc.js` + `docs/status.generated.md`** — single-
|
|
47
|
+
source-of-truth status snapshot that pulls live values from `package.json`,
|
|
48
|
+
the schema, the manifest, and the four deterministic check scripts.
|
|
49
|
+
Replaces hand-maintained "Latest release" lines and ad-hoc skill-count
|
|
50
|
+
claims. Flags: `--check`, `--stdout`, `--no-checks`.
|
|
51
|
+
- **`skill-graph doctor`** — new CLI subcommand grouping all six deterministic
|
|
52
|
+
checks (links, protocol, drift, mirror-freeze, lint, manifest) into one
|
|
53
|
+
pass with a single summary table. Recommended first command for bug
|
|
54
|
+
reports. Flags: `--json`, `--bail`, `--skip <name>`.
|
|
55
|
+
- **`examples/fixture-skills/`** — hermetic test fixtures for `@skill-graph/cli`
|
|
56
|
+
package tests. Ships `minimal-capability` (bare-minimum v6 frontmatter,
|
|
57
|
+
passes lint with zero errors). Three more planned (`with-grounding`,
|
|
58
|
+
`with-relations`, `comprehension-full`) are sketched in the directory
|
|
59
|
+
README; each will land as a separate commit when the sub-field shapes
|
|
60
|
+
are aligned with the lint's expectations.
|
|
61
|
+
- **`docs/positioning.md`** — explicit positioning of Skill Graph in the
|
|
62
|
+
agent-skills ecosystem. Names what Skill Graph is **not** (a runtime,
|
|
63
|
+
a marketplace, a tool platform) before naming what it **is** (an
|
|
64
|
+
authoring + audit-time contract with a Karpathy keep-or-revert loop).
|
|
65
|
+
Compares against Anthropic Skills, the Agent Skills spec, MCP, A2A,
|
|
66
|
+
Smithery, Composio, and AGENTS.md. Includes the "60-second pitch",
|
|
67
|
+
the "what this is not" matrix, the "when to reach for Skill Graph"
|
|
68
|
+
gate, and the layered ecosystem map.
|
|
69
|
+
- **README hero rework** — three new sections above the ecosystem
|
|
70
|
+
diagram: "Is this for me?" (yes-if / no-if qualifying signals for a
|
|
71
|
+
30-second visitor), "What makes this different" (Karpathy keep-or-revert
|
|
72
|
+
loop applied to skill libraries, with explicit citation), and a
|
|
73
|
+
surfacing of `docs/quality-doctrine.md` as the codified quality bar.
|
|
74
|
+
- **README two-onboarding-paths framing** — explicit routing of authors
|
|
75
|
+
to `docs/QUICKSTART-30MIN.md` for hands-on first-skill authoring and
|
|
76
|
+
to `docs/PRIMER.md` for the conceptual model first. Synthesis Item 48
|
|
77
|
+
resolved as "link, do not merge" because they serve different reader
|
|
78
|
+
genres.
|
|
79
|
+
|
|
80
|
+
### Changed
|
|
81
|
+
|
|
82
|
+
- **`scripts/check-markdown-links.js`** — broken links in `_archived/`
|
|
83
|
+
paths now emit warnings instead of failing the build. Active docs
|
|
84
|
+
remain strict. New `--strict-archived` flag elevates archived warnings
|
|
85
|
+
to errors (useful when migrating an archive batch). Synthesis Item 49.
|
|
86
|
+
- **README "External Context"** — replaced 5-bullet list of tangential
|
|
87
|
+
references with a "Where Skill Graph fits" section that links directly
|
|
88
|
+
to `docs/positioning.md` and surfaces the unique angle in one line.
|
|
89
|
+
- **`bin/skill-graph.js`** — `evolve` subcommand labeled `[PREVIEW ·
|
|
90
|
+
monorepo-only]` in both the README quickstart and the CLI `--help`.
|
|
91
|
+
Aligns the discoverability surface with the dispatcher's existing
|
|
92
|
+
not-standalone-compatible error message.
|
|
93
|
+
|
|
94
|
+
### Documented
|
|
95
|
+
|
|
96
|
+
- **`docs/_archived/marketplace-publication-priority-2026-05-18.md`** —
|
|
97
|
+
the archived plan's broken `./adr/...` link was fixed to `../adr/...`
|
|
98
|
+
(Phase 1 Item 1).
|
|
99
|
+
- **Mirror governance** — five mirror docs banner-stamped as frozen
|
|
100
|
+
deprecation snapshots: `skill-metadata-protocol/docs/{concept-map,
|
|
101
|
+
field-reference, field-reference.generated, skill-metadata-protocol}.md`
|
|
102
|
+
and `skill-metadata-protocol/CONTRIBUTING.md` + `skill-audit-loop/{
|
|
103
|
+
CONTRIBUTING, SKILL_AUDIT_LOOP, SKILL_AUDIT_CHECKLIST}.md` (Phase 1
|
|
104
|
+
Items 30, 25 + Phase 2 Item 36 follow-ups).
|
|
105
|
+
- **`docs/plans/skill-graph-oss-docs-refresh.md`** (lives in the
|
|
106
|
+
Development workspace, not this repo) — banner-stamped as superseded
|
|
107
|
+
by ADR 0009 (Phase 2 Item 41).
|
|
108
|
+
|
|
109
|
+
### Carried forward from earlier [Unreleased]
|
|
110
|
+
|
|
111
|
+
[Move the existing pre-Phase-2 [Unreleased] entries here — i.e. the
|
|
112
|
+
publication-classification ledger, the publish.yml workflow, the
|
|
113
|
+
stability promotion lint check, the marketplace publication priority
|
|
114
|
+
doc removal, the cross-repo version reconciliation, the schema_version 4/5/6
|
|
115
|
+
changes, and the skill audit loop reframing.]
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
## Apply the diff (when ready)
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
cd /Users/jacobbalslev/Development/skill-graph
|
|
122
|
+
|
|
123
|
+
# 1. Bump version
|
|
124
|
+
sed -i '' 's/"version": "0.5.7"/"version": "0.5.8"/' package.json
|
|
125
|
+
|
|
126
|
+
# 2. Hand-edit CHANGELOG.md per the proposed entry above. The mechanical
|
|
127
|
+
# steps are:
|
|
128
|
+
# - Move existing [Unreleased] body to "## [0.5.8] — 2026-05-19" with
|
|
129
|
+
# a "Carried forward from earlier [Unreleased]" sub-section.
|
|
130
|
+
# - Add the new Phase 2 entries from this draft.
|
|
131
|
+
# - Insert a fresh empty "## [Unreleased]" header above [0.5.8].
|
|
132
|
+
|
|
133
|
+
# 3. Regenerate status doc so it reflects the new version
|
|
134
|
+
node scripts/build-status-doc.js
|
|
135
|
+
|
|
136
|
+
# 4. Verify everything's clean
|
|
137
|
+
node bin/skill-graph.js doctor
|
|
138
|
+
|
|
139
|
+
# 5. Commit with --only
|
|
140
|
+
git commit --only -m "chore(release): cut 0.5.8 — Karpathy-loop Phase 2
|
|
141
|
+
|
|
142
|
+
Promotes [Unreleased] to [0.5.8] and adds Phase 2 entries:
|
|
143
|
+
drift sentinel, mirror-freeze linter, status doc, doctor subcommand,
|
|
144
|
+
fixture-skills, positioning doc, README hero rework, evolve PREVIEW
|
|
145
|
+
labeling, and the _archived/ link-check policy.
|
|
146
|
+
|
|
147
|
+
Karpathy-loop Phase 2 item 34/55." -- package.json CHANGELOG.md docs/status.generated.md
|
|
148
|
+
|
|
149
|
+
# 6. Tag + publish (only after committing and pushing)
|
|
150
|
+
git tag v0.5.8
|
|
151
|
+
git push origin main --tags
|
|
152
|
+
# CI workflow .github/workflows/publish.yml publishes on tag push.
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
## Open questions for the user
|
|
156
|
+
|
|
157
|
+
1. Should I keep all pre-Phase-2 [Unreleased] entries inside [0.5.8] (the
|
|
158
|
+
"kitchen-sink" choice), or split them into a separate intermediate version
|
|
159
|
+
(e.g. 0.5.6 / 0.5.7 / 0.5.8) to chronicle when they actually shipped?
|
|
160
|
+
The package.json version has already bumped 0.5.0 → 0.5.7 without
|
|
161
|
+
corresponding CHANGELOG entries, so there's accumulated drift.
|
|
162
|
+
2. Confirm the release tag will trigger `.github/workflows/publish.yml`.
|
|
163
|
+
First-publish prereqs (npm org + `NPM_TOKEN` secret) are tracked in the
|
|
164
|
+
existing `## [Unreleased]` SH-6111 entry.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Skill Graph Field Reference (Generated)
|
|
2
2
|
|
|
3
|
-
> **Generated from** `schemas/skill.schema.json` on 2026-05-
|
|
3
|
+
> **Generated from** `schemas/skill.schema.json` on 2026-05-19 by `scripts/build-field-reference.js`.
|
|
4
4
|
> **Do not edit by hand.** The canonical prose reference is [`docs/field-reference.md`](field-reference.md).
|
|
5
5
|
> **Predicate glossary:** [`docs/glossary.md`](glossary.md).
|
|
6
6
|
> **JSON-LD @context:** [`schemas/skill.context.jsonld`](../schemas/skill.context.jsonld).
|
package/docs/field-reference.md
CHANGED
|
@@ -13,7 +13,7 @@ The field reference is split across three coordinated documents. Use whichever f
|
|
|
13
13
|
| Doc | Genre | When to read |
|
|
14
14
|
|---|---|---|
|
|
15
15
|
| [`field-reference.md`](field-reference.md) (this doc) | **Hand-curated prose reference.** Field-by-field, with worked examples, lint notes, and cross-cutting guidance. | When authoring or reviewing a SKILL.md and you want examples and "when to use" rules alongside the schema-canonical definition. |
|
|
16
|
-
| [`field-reference.generated.md`](field-reference.generated.md) | **Auto-generated index.** Built from `schemas/skill.
|
|
16
|
+
| [`field-reference.generated.md`](field-reference.generated.md) | **Auto-generated index.** Built from `schemas/skill.v6.schema.json` description strings by `scripts/build-field-reference.js`. Drift-free against the schema. | When you want the machine-guaranteed list of every field, every type, every pattern, every enum value. The fastest way to verify what the schema actually accepts today. |
|
|
17
17
|
| [`field-rationale.md`](field-rationale.md) | **Hand-authored "why this field" rationale.** Covers the ~10 fields whose meaning is non-obvious from the schema description (`scope`, `eval_artifacts`, `eval_state`, `routing_eval`, `relations.depends_on`, `relations.verify_with`, `relations.broader`, `grounding.evidence_priority`, `lifecycle.review_cadence`, `portability.readiness`). | When you understand *what* a field stores but want to know *why the field exists at all* and *what the common confusion looks like*. |
|
|
18
18
|
|
|
19
19
|
The schema is the single source of truth for shape; this doc is the source of truth for prose; `field-rationale.md` is the source of truth for design intent. Lint check C7 (in `scripts/check-protocol-consistency.js`) verifies the generated index stays in sync with the schema description strings — running `node scripts/build-field-reference.js --check` against the live schema must succeed before commit.
|
|
@@ -34,7 +34,7 @@ The schema is the single source of truth for shape; this doc is the source of tr
|
|
|
34
34
|
|
|
35
35
|
**Example.**
|
|
36
36
|
```yaml
|
|
37
|
-
schema_version:
|
|
37
|
+
schema_version: 6
|
|
38
38
|
```
|
|
39
39
|
|
|
40
40
|
**When to use.** Always — this is a required field.
|
|
@@ -148,8 +148,8 @@ Three versions coexist in a manifest ecosystem:
|
|
|
148
148
|
| Version | Lives in | Meaning |
|
|
149
149
|
|---|---|---|
|
|
150
150
|
| Authored skill `version` | Per-skill frontmatter `version` field | Version of the skill's content (e.g. `1.2.0` means the skill has been iterated twice since its initial publish). |
|
|
151
|
-
| Authored schema version | Per-skill frontmatter `schema_version` field | Version of the `skill.schema.json` contract the skill was authored against.
|
|
152
|
-
| Manifest schema version | Manifest root `schema_version` field | Version of the `manifest.schema.json` contract the manifest was generated against.
|
|
151
|
+
| Authored skill schema version | Per-skill frontmatter `schema_version` field | Version of the `skill.schema.json` contract the skill was authored against. The active value is `6` (skill schema, advanced through v5 to v6 marketplace fields). |
|
|
152
|
+
| Manifest schema version | Manifest root `schema_version` field | Version of the `manifest.schema.json` contract the manifest was generated against. The active value is `4` — manifest contract shape has not changed since v4 even though skill schema has advanced. |
|
|
153
153
|
|
|
154
154
|
### When to bump `schema_version`
|
|
155
155
|
|
|
@@ -302,7 +302,7 @@ The `skill-metadata-template` starter (`examples/skill-metadata-template.md`) is
|
|
|
302
302
|
|
|
303
303
|
```yaml
|
|
304
304
|
---
|
|
305
|
-
schema_version:
|
|
305
|
+
schema_version: 6
|
|
306
306
|
name: skill-metadata-template
|
|
307
307
|
description: "Authoring template for new Skill Metadata Protocol skills. ..."
|
|
308
308
|
version: 1.0.0
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Marketplace Publication Queue (generated)
|
|
2
2
|
|
|
3
|
-
Generated: 2026-05-
|
|
3
|
+
Generated: 2026-05-18T13:56:43.757Z
|
|
4
4
|
Schema: 2.0.0
|
|
5
5
|
Generator: skill-audit-loop@0.3.0
|
|
6
6
|
Source ledger: `skill-graph/data/publication-classification.json`
|
|
@@ -14,7 +14,7 @@ Source ledger: `skill-graph/data/publication-classification.json`
|
|
|
14
14
|
- Publishable candidates: 136
|
|
15
15
|
- Sales-Hub-bound (excluded): 53
|
|
16
16
|
- Personal-infra (excluded): 27
|
|
17
|
-
- Already published (in OSS export):
|
|
17
|
+
- Already published (in OSS export): 142
|
|
18
18
|
- Tier counts: S=18 A=29 B=42 C=47
|
|
19
19
|
- Ledger entries total: 216
|
|
20
20
|
|
|
@@ -77,14 +77,14 @@ Purpose:
|
|
|
77
77
|
Shipping today. Two modes:
|
|
78
78
|
|
|
79
79
|
1. **Stub mode** (default) — runs `scripts/skill-lint.js` and seeds `audits/<skill>/{findings,verdict,scorecard}.md` with lint-derived findings and human-TODO placeholders for the seven qualitative dimensions.
|
|
80
|
-
2. **`--graded` mode** — on top of the stub, composes per-dimension prompts via `scripts/lib/audit-prompt-builder.js` and calls an external
|
|
80
|
+
2. **`--graded` mode** — on top of the stub, composes per-dimension prompts via `scripts/lib/audit-prompt-builder.js` and calls an explicit external grader CLI (`--grader-cli "<command>"`) once per dimension. Each response must return a `<verdict>…</verdict>` JSON block matching the fixed schema (dimension / score / verdict / justification / findings[]). The runner parses, validates, and merges these into the artifact files, replacing TODO placeholders with structured PASS / PASS WITH FIXES / FAIL verdicts with evidence quotes.
|
|
81
81
|
|
|
82
82
|
Related files:
|
|
83
83
|
|
|
84
84
|
- `scripts/lib/audit-prompt-builder.js` — dimension registry, context collector, prompt composer, response parser, verdict aggregator
|
|
85
85
|
- `scripts/lib/mock-grader.js` — deterministic canned-response grader for CI smoke-tests and reproducible examples (prints fixed `<verdict>` blocks per dimension)
|
|
86
86
|
|
|
87
|
-
Grader CLI discipline: the runner NEVER embeds API keys
|
|
87
|
+
Grader CLI discipline: the runner NEVER embeds API keys, never selects a default provider, and never writes model-routing metadata into Skill Graph project artifacts. Auth and provider choice are delegated to the explicit external CLI command supplied by the caller.
|
|
88
88
|
|
|
89
89
|
## Suggested Follow-on Scripts
|
|
90
90
|
|
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
# Positioning — Skill Graph in the Agent-Skills Ecosystem
|
|
2
|
+
|
|
3
|
+
> **One-line:** Skill Graph is the **authoring contract + audit loop** for agent skills — it sits *above* plain `SKILL.md` files and *beside* protocols like MCP and A2A. It does not host, dispatch, or execute skills.
|
|
4
|
+
|
|
5
|
+
## What this answers
|
|
6
|
+
|
|
7
|
+
If you came in via the README hero and asked "where does this fit?", this doc names every adjacent project and draws the boundary. The goal is to make the
|
|
8
|
+
unique angle of Skill Graph legible in under 60 seconds.
|
|
9
|
+
|
|
10
|
+
## The 60-second pitch
|
|
11
|
+
|
|
12
|
+
Most agent-skill projects answer one of these questions:
|
|
13
|
+
|
|
14
|
+
1. **How does an agent find and load a skill?** (Anthropic Skills, agent-skills registries)
|
|
15
|
+
2. **How does an agent call a tool?** (MCP)
|
|
16
|
+
3. **How do agents talk to each other?** (A2A)
|
|
17
|
+
4. **Where do skills get packaged and discovered?** (Smithery, marketplaces)
|
|
18
|
+
5. **How does an agent author orchestrate skill calls in production?** (Composio, agent platforms)
|
|
19
|
+
|
|
20
|
+
Skill Graph answers a **different** question: **how do you keep a library of skills correct over time?**
|
|
21
|
+
|
|
22
|
+
It does this by giving every skill a structured contract (`SKILL.md` frontmatter), a graph of typed edges between skills (`relations.*`), a deterministic audit loop (`audit → improve → evaluate`), and a Karpathy-style keep-or-revert discipline for changes. The contract is portable; the audit loop is the unique mechanism.
|
|
23
|
+
|
|
24
|
+
## The "what this is not" matrix
|
|
25
|
+
|
|
26
|
+
Skill Graph is not a competitor to MCP, A2A, Anthropic Skills, Smithery, or Composio. They sit at different layers:
|
|
27
|
+
|
|
28
|
+
| Project | What it is | What Skill Graph is in that frame |
|
|
29
|
+
|---|---|---|
|
|
30
|
+
| **[Anthropic Skills](https://docs.anthropic.com/en/docs/claude-code/skills)** | A discovery + loading convention: agent-runtime auto-loads `SKILL.md` files from a directory and decides what to invoke. | Skill Graph **extends** the `SKILL.md` shape with structured metadata (relations, grounding, eval health) so a library of 100+ skills stays coherent. A plain `SKILL.md` still loads everywhere; Skill Graph adds validation, drift checks, and graph queries on top. |
|
|
31
|
+
| **[Agent Skills spec](https://agentskills.my/specification)** | A portable `SKILL.md` packaging format with a draft cross-runtime contract. | Skill Graph **emits** Agent-Skills-compatible exports (see [`skill-graph export`](../README.md#quick-start)). The Skill Metadata Protocol adds typed relations and audit fields the base spec does not require, but the export is plain. |
|
|
32
|
+
| **[Model Context Protocol (MCP)](https://modelcontextprotocol.io)** | A *runtime* protocol: how an agent client calls a tool server (function calling, structured tools, resources, prompts). | Skill Graph is **build-time and authoring-time**, not runtime. Skills can describe how an agent should use an MCP server (in their body or `relations.depends_on`), but Skill Graph itself does not implement MCP. |
|
|
33
|
+
| **[Agent-to-Agent (A2A)](https://google.github.io/A2A/latest/specification/)** | A *runtime* protocol: how one agent delegates a task to another agent and exchanges capability cards. | Skill Graph's metadata could serve as the capability descriptor an A2A agent advertises, but Skill Graph does not implement A2A delegation, transport, or task lifecycle. |
|
|
34
|
+
| **[Smithery](https://smithery.ai)** | A *registry*: searchable directory of MCP servers and agent skills with install confidence and compatibility signals. | Skill Graph's marketplace export pipeline (`marketplace/skills/`) is one upstream that *feeds* a registry like Smithery; Skill Graph itself is the authoring source, not the registry. |
|
|
35
|
+
| **[Composio](https://docs.composio.dev)** | An *agent tooling platform*: hosted integrations, auth, and tool execution for production agents. | Skill Graph and Composio operate at different layers entirely. A Composio agent could include Skill-Graph-published skills as part of its task brief; Skill Graph does not host or dispatch. |
|
|
36
|
+
| **[AGENTS.md](https://agents.md)** | A *repository-local* convention for instructing coding agents (Claude, Cursor, Codex, etc.) on how to work in a specific repo. | Skill Graph can **ingest** an `AGENTS.md` as a context source for a codebase-scope skill's `grounding` field, but it does not compete with the convention. Skill Graph is repository-portable; AGENTS.md is repository-local. |
|
|
37
|
+
|
|
38
|
+
## The Karpathy axis (what makes Skill Graph itself distinct)
|
|
39
|
+
|
|
40
|
+
Most of the above projects are concerned with the *what* and *how* of agent skills. Skill Graph adds an **opinionated discipline** for the *when* — when a skill is changed:
|
|
41
|
+
|
|
42
|
+
- **One field, one commit, one keep-or-revert decision** ([Karpathy's autoresearch](https://github.com/karpathy/autoresearch) loop, applied to skill libraries instead of training scripts).
|
|
43
|
+
- Every change has a hard pass/fail gate (a deterministic check script that turns red or green).
|
|
44
|
+
- Failed changes auto-revert. The lesson is recorded; the field's truth is preserved.
|
|
45
|
+
|
|
46
|
+
That discipline is the unique angle. The metadata schema is the substrate that makes it possible — without typed fields, you can't have a deterministic gate. The audit loop is the mechanism. The result is a skill library that drifts less, even as it grows.
|
|
47
|
+
|
|
48
|
+
## When to reach for Skill Graph
|
|
49
|
+
|
|
50
|
+
Reach for it when:
|
|
51
|
+
|
|
52
|
+
- You have **more than ~5 skills** and they have started to depend on, verify, or exclude one another.
|
|
53
|
+
- You want **deterministic checks** for skill correctness (schema, paths, eval health) and not just LLM-as-grader.
|
|
54
|
+
- You want **graph queries** over the library — "what skills depend on this one?", "what's the boundary between X and Y?", "which skills verify this one?"
|
|
55
|
+
- You want a **single audit loop** that produces a fingerprint per skill (`audit_verdict`, `eval_score`, `drift_status`) you can ship in the skill's own frontmatter.
|
|
56
|
+
|
|
57
|
+
Do NOT reach for it when:
|
|
58
|
+
|
|
59
|
+
- You have 1–3 skills and a plain folder is enough.
|
|
60
|
+
- You want a hosted skill marketplace (use Smithery, agentskills.io).
|
|
61
|
+
- You want an agent runtime (use Claude Code, Cursor, Codex, etc.).
|
|
62
|
+
- You want a tool execution platform (use Composio, your agent runtime's tool layer).
|
|
63
|
+
|
|
64
|
+
## Where this puts Skill Graph
|
|
65
|
+
|
|
66
|
+
In the ecosystem map, Skill Graph sits in the **author-time + audit-time** layer:
|
|
67
|
+
|
|
68
|
+
```
|
|
69
|
+
runtime layer Anthropic Skills | MCP | A2A | Composio | Smithery
|
|
70
|
+
↑ ↑ ↑ ↑ ↑
|
|
71
|
+
| | | | | (consumes published skills)
|
|
72
|
+
author/audit-time layer └───┴─────┴───────┴──────────┘
|
|
73
|
+
↑
|
|
74
|
+
┌───────────┴────────────┐
|
|
75
|
+
| Skill Graph (here) |
|
|
76
|
+
| + Karpathy audit loop |
|
|
77
|
+
└────────────────────────┘
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
The runtime layer answers "how does this skill execute?" The author/audit-time layer answers "how does this library stay correct?" Both are needed; both are different products.
|
|
81
|
+
|
|
82
|
+
## Related reading
|
|
83
|
+
|
|
84
|
+
- [`docs/SKILL_METADATA_PROTOCOL.md`](SKILL_METADATA_PROTOCOL.md) — the contract.
|
|
85
|
+
- [`SKILL_AUDIT_LOOP.md`](SKILL_AUDIT_LOOP.md) — the four operations (audit, improve, evaluate, evolve).
|
|
86
|
+
- [`docs/quality-doctrine.md`](quality-doctrine.md) — what "improve" means in this discipline.
|
|
87
|
+
- [`docs/adr/0009-sibling-repo-deprecation.md`](adr/0009-sibling-repo-deprecation.md) — why the protocol, audit, and CLI live in one repo now.
|
|
88
|
+
- [Karpathy autoresearch](https://github.com/karpathy/autoresearch) — the keep-or-revert loop applied to LLM training scripts that Skill Graph borrows for skill libraries.
|
|
@@ -53,7 +53,7 @@ The report does not propose a new schema version, does not redesign lint or drif
|
|
|
53
53
|
|
|
54
54
|
## 2. What the repo currently evaluates
|
|
55
55
|
|
|
56
|
-
The Skill Graph's evaluation discipline is described in [`AGENTS.
|
|
56
|
+
The Skill Graph's evaluation discipline is described in [`AGENTS.md` § Evaluation Discipline (lines 143–189)](/Users/jacobbalslev/Development/skill-graph/AGENTS.md). It names four layers, each with its own definition of "good":
|
|
57
57
|
|
|
58
58
|
| Layer | Question | Surface | Deterministic? |
|
|
59
59
|
|---|---|---|---|
|
|
@@ -171,7 +171,7 @@ The result: a skill author who fills the `concept` block does not know what a co
|
|
|
171
171
|
|
|
172
172
|
### 3.2 No verbatim-copy detector
|
|
173
173
|
|
|
174
|
-
A failure mode the protocol's quality doctrine warns against — "evals that paraphrase the skill body back to itself" ([`AGENTS.
|
|
174
|
+
A failure mode the protocol's quality doctrine warns against — "evals that paraphrase the skill body back to itself" ([`AGENTS.md` line 186](/Users/jacobbalslev/Development/skill-graph/AGENTS.md)) — has no test. The current eval files frequently use prompts like:
|
|
175
175
|
|
|
176
176
|
> "According to the X skill's Y section, what is the correct primitive…"
|
|
177
177
|
> (e.g. [`examples/evals/a11y.json:8-16`](/Users/jacobbalslev/Development/skill-graph/examples/evals/a11y.json))
|
|
@@ -1045,7 +1045,7 @@ The 10 cases cover all 9 rubric dimensions:
|
|
|
1045
1045
|
|
|
1046
1046
|
Transfer mix: 6 near-transfer (C1, C3, C4, C7-#1, C8, C9), 4 far-transfer (C2, C5, C6, C7-#2). The schema's heaviest-weighted dimensions (mental_model 1.5, boundary 1.5) both have far-transfer coverage. Misconception has two cases — one near (the Zod question, a domain-internal misconception) and one far (the C++ comparison, a cross-language misconception), exercising the misconception primitive across surface variations.
|
|
1047
1047
|
|
|
1048
|
-
The minimum threshold (`AGENTS.
|
|
1048
|
+
The minimum threshold (`AGENTS.md § Evaluation Discipline`: "≥7 realistic scenarios per skill") is exceeded; this file would also satisfy the existing audit-loop's Eval dimension if the grader is updated to accept the comprehension-extension keys.
|
|
1049
1049
|
|
|
1050
1050
|
### 6.4 What this worked example does NOT cover
|
|
1051
1051
|
|
|
@@ -1169,7 +1169,7 @@ Update [`SKILL_AUDIT_CHECKLIST.md`](https://github.com/jacob-balslev/skill-audit
|
|
|
1169
1169
|
|
|
1170
1170
|
**Effort.** ~half day.
|
|
1171
1171
|
|
|
1172
|
-
**Change.** Add a new section to `docs/skill-metadata-protocol.md` and `docs/field-reference.md § concept` explaining the rubric. Cross-reference from `AGENTS.
|
|
1172
|
+
**Change.** Add a new section to `docs/skill-metadata-protocol.md` and `docs/field-reference.md § concept` explaining the rubric. Cross-reference from `AGENTS.md § Evaluation Discipline` so the protocol-level discipline is discoverable. Add the new optional eval-file keys (`comprehension_dimension`, `concept_field`, `transfer`, `expected_behaviors`) to `docs/field-reference.md` with the same field-level treatment as existing keys.
|
|
1173
1173
|
|
|
1174
1174
|
**Why this is leverage-8.** Without documentation, the rubric is folklore. With documentation in the canonical reference doc, authors discover it during their normal skill-authoring flow.
|
|
1175
1175
|
|
|
@@ -1397,7 +1397,7 @@ Recommendation, not decision: defer until R3 ships. Once the grader runs and pro
|
|
|
1397
1397
|
|
|
1398
1398
|
This report examined the following **24 repo files** (23 inside `skill-graph/` plus 1 in the parent `Development/skills/` workspace, added in the 2026-05-16 revision):
|
|
1399
1399
|
|
|
1400
|
-
1. [`/Users/jacobbalslev/Development/skill-graph/AGENTS.
|
|
1400
|
+
1. [`/Users/jacobbalslev/Development/skill-graph/AGENTS.md`](/Users/jacobbalslev/Development/skill-graph/AGENTS.md) — full read; especially §§ Evaluation Discipline, Skill Audit Loop, What the Skill Graph Is
|
|
1401
1401
|
2. [`https://github.com/jacob-balslev/skill-metadata-protocol`](https://github.com/jacob-balslev/skill-metadata-protocol) — full read
|
|
1402
1402
|
3. [`https://github.com/jacob-balslev/skill-audit-loop/blob/main/SKILL_AUDIT_LOOP.md`](https://github.com/jacob-balslev/skill-audit-loop/blob/main/SKILL_AUDIT_LOOP.md) — full read
|
|
1403
1403
|
4. [`https://github.com/jacob-balslev/skill-audit-loop/blob/main/SKILL_AUDIT_CHECKLIST.md`](https://github.com/jacob-balslev/skill-audit-loop/blob/main/SKILL_AUDIT_CHECKLIST.md) — full read
|