bigpowers 2.63.0 → 2.64.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.pi/package.json +1 -1
- package/.pi/prompts/build-epic.md +2 -1
- package/.pi/prompts/develop-tdd.md +4 -0
- package/.pi/prompts/investigate-bug.md +5 -0
- package/.pi/prompts/plan-refactor.md +5 -0
- package/.pi/prompts/run-benchmark.md +67 -48
- package/.pi/prompts/security-review.md +4 -0
- package/.pi/prompts/setup-environment.md +15 -0
- package/.pi/prompts/wire-observability.md +4 -0
- package/.pi/skills/build-epic/SKILL.md +2 -1
- package/.pi/skills/develop-tdd/SKILL.md +4 -0
- package/.pi/skills/investigate-bug/SKILL.md +5 -0
- package/.pi/skills/plan-refactor/SKILL.md +5 -0
- package/.pi/skills/run-benchmark/SKILL.md +67 -48
- package/.pi/skills/security-review/SKILL.md +4 -0
- package/.pi/skills/setup-environment/SKILL.md +15 -0
- package/.pi/skills/wire-observability/SKILL.md +4 -0
- package/CHANGELOG.md +25 -0
- package/CLAUDE.md +4 -0
- package/CONVENTIONS.md +10 -0
- package/README.md +6 -0
- package/SKILL-INDEX.md +1 -1
- package/package.json +5 -2
- package/scripts/build-receipts.sh +107 -0
- package/scripts/build-skill-graph.sh +226 -0
- package/scripts/build-skill-index.sh +10 -20
- package/scripts/generate-skill-index.sh +2 -4
- package/scripts/record-cycle-time.sh +6 -4
- package/scripts/regenerate-lockfile.sh +10 -38
- package/scripts/sync-skills.sh +106 -79
- package/scripts/validate-okf.sh +28 -0
- package/skills/build-epic/SKILL.md +2 -1
- package/skills/develop-tdd/SKILL.md +4 -0
- package/skills/investigate-bug/SKILL.md +5 -0
- package/skills/plan-refactor/SKILL.md +5 -0
- package/skills/run-benchmark/SKILL.md +67 -49
- package/skills/security-review/SKILL.md +4 -0
- package/skills/setup-environment/SKILL.md +15 -0
- package/skills/wire-observability/SKILL.md +4 -0
- package/skills-lock.json +83 -83
package/.pi/package.json
CHANGED
|
@@ -33,7 +33,8 @@ Orchestrates the **build** flow for a single epic: survey → plan tasks → kic
|
|
|
33
33
|
2. **Step 0 — Threat Model:** Run `security-review` against the epic's scope (read from the epic capsule). Output `specs/security/epics/<epic-id>/THREAT_MODEL.md` with surface area, vulnerability categories, risk level, and mitigation guidance.
|
|
34
34
|
3. **Assess Impact (Step 2):** Before writing tasks, run `assess-impact --lightweight` on the proposed change. If the risk score exceeds 7, gate — require a `grill-me` session. Write the impact report to `specs/IMPACT-<epic>-<story>.md`. For net-new code with no existing dependents, skip.
|
|
35
35
|
4. **BCP Tracking (Step 2):** After `plan-work` completes, read the `bcps:` count (Business Complexity Points story size) from the epic capsule and carry it into `state.yaml` as `epic_cycle.story_bcps = N`.
|
|
36
|
-
5.
|
|
36
|
+
5. **BCP Plus (Step 2, optional):** When the epic capsule carries a `bcp_plus_breakdown` (13-dimension sizing), also carry it into `state.yaml` as `epic_cycle.bcp_plus`. The breakdown maps each of the 13 dimensions to an integer count. NFR dimensions (11–13) that are N/A per the NFR Gate may be set to 0.
|
|
37
|
+
6. If `epic_cycle.step` missing, set to `1`.
|
|
37
38
|
6. Run **only the current step** (resume mode) unless user asked for full auto-run.
|
|
38
39
|
7. After step verify passes, increment `epic_cycle.step` in `state.yaml` (or `bash scripts/bp-yaml-set.sh` if available).
|
|
39
40
|
8. On story complete, set `execution-status.yaml` story key to `done`; run `bash scripts/sync-status-from-epics.sh`.
|
|
@@ -101,6 +101,10 @@ If this cycle modified files in `.github/workflows/`, run the CI dry-run procedu
|
|
|
101
101
|
Gate: READY -> next: verify-work
|
|
102
102
|
Writes: state.yaml handoff.next_skill = verify-work
|
|
103
103
|
|
|
104
|
+
## BCP Plus Integration
|
|
105
|
+
|
|
106
|
+
At story completion, if the story was sized with BCP Plus (13-dimension breakdown), log the `bcp_plus.total` alongside the standard `bcps:` count in the story's tasks.yaml. The breakdown is available from the epic capsule's `bcp_plus_breakdown` field. See `docs/references/bcp-plus.md` for the full methodology and NFR Gate pattern.
|
|
107
|
+
|
|
104
108
|
---
|
|
105
109
|
|
|
106
110
|
# Develop TDD — Reference
|
|
@@ -117,3 +117,8 @@ A numbered list of RED-GREEN cycles:
|
|
|
117
117
|
</diagnosis-template>
|
|
118
118
|
|
|
119
119
|
After writing the bug file, print a one-line summary of the root cause and suggest running `kickoff-branch` next to create a fix branch.
|
|
120
|
+
|
|
121
|
+
## References
|
|
122
|
+
|
|
123
|
+
- [Feathers' Seams & Characterization Tests](../../docs/references/feathers.md) — seam types, characterization tests, and the legacy code change algorithm
|
|
124
|
+
- [Fowler's Code Smells](../../docs/references/fowler.md) — identifying structural problems by smell before investigating
|
|
@@ -76,4 +76,9 @@ Any further notes about the refactor.
|
|
|
76
76
|
|
|
77
77
|
After writing `specs/REFACTOR_LATEST.md`, suggest running `kickoff-branch` next to create a refactor branch.
|
|
78
78
|
|
|
79
|
+
## References
|
|
80
|
+
|
|
81
|
+
- [Fowler's Refactoring Catalog](../../docs/references/fowler.md) — canonical refactoring vocabulary and code-smell taxonomy
|
|
82
|
+
- [Beck's Tidy First?](../../docs/references/kent-beck.md) — structural change before behavioral change
|
|
83
|
+
|
|
79
84
|
## Verify
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
---
|
|
2
|
-
description: Run skill quality benchmarks from specs/benchmarks/ definitions
|
|
2
|
+
description: Run skill quality benchmarks from specs/benchmarks/ definitions — N-run with/without-skill delta grading, train/validation split, pass@k + benchmark.json reports. Use before and after evolve-skill to prove quality changes are improvements, not regressions.
|
|
3
3
|
---
|
|
4
4
|
|
|
5
5
|
|
|
@@ -7,59 +7,78 @@ description: Run skill quality benchmarks from specs/benchmarks/ definitions and
|
|
|
7
7
|
|
|
8
8
|
> **HARD GATE** — Do NOT use benchmark scores to declare a skill "good" or "bad" in isolation. Benchmarks measure relative quality vs. a baseline — they catch regressions, they do not certify correctness.
|
|
9
9
|
|
|
10
|
-
Reads benchmark definitions from `specs/benchmarks/`, executes each scenario's grader, and writes a structured `pass@k` report that `evolve-skill` consumes.
|
|
10
|
+
Reads benchmark definitions from `specs/benchmarks/`, executes each scenario's grader with and without the skill loaded, and writes a structured `pass@k` report with delta grading that `evolve-skill` consumes.
|
|
11
11
|
|
|
12
|
-
##
|
|
12
|
+
## With/Without-Skill Delta Grading
|
|
13
13
|
|
|
14
|
-
|
|
15
|
-
# Benchmark a single skill
|
|
16
|
-
run-benchmark <skill-name>
|
|
14
|
+
Every scenario runs N times (default 3) in two modes: **with** the skill loaded and **without** (bare agent with only CLAUDE.md). The delta `Δ = pass@k_with − pass@k_without` isolates the skill's causal contribution. A negative delta is a regression flag.
|
|
17
15
|
|
|
18
|
-
|
|
19
|
-
run-benchmark --all
|
|
16
|
+
## Train/Validation Split
|
|
20
17
|
|
|
21
|
-
|
|
22
|
-
run-benchmark <skill-name> --baseline
|
|
23
|
-
```
|
|
18
|
+
Benchmark definitions partition scenarios into two sets:
|
|
24
19
|
|
|
25
|
-
|
|
20
|
+
| Set | Tag | Purpose |
|
|
21
|
+
|-----|-----|---------|
|
|
22
|
+
| **Train** | `split: train` | Development scenarios — used while iterating. Hitting 100% on train is expected. |
|
|
23
|
+
| **Validation** | `split: validation` | Held-out scenarios — the real quality signal. Overfitting train while validation stagnates is a design smell. |
|
|
24
|
+
|
|
25
|
+
`pass@k` is reported separately for train and validation. Validation score is authoritative; train score is iteration guidance only.
|
|
26
26
|
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
3. **Calculate pass@k** — `pass@k = sum(weight of PASS scenarios) / sum(all weights)`. Round to 2 decimal places.
|
|
34
|
-
|
|
35
|
-
4. **Write report** to `specs/benchmarks/reports/BENCHMARK-<skill>-<YYYY-MM-DD>.yaml`:
|
|
36
|
-
|
|
37
|
-
```yaml
|
|
38
|
-
skill: survey-context
|
|
39
|
-
run_date: "2026-06-22"
|
|
40
|
-
pass_at_k: 0.83
|
|
41
|
-
total_scenarios: 3
|
|
42
|
-
passed: 2
|
|
43
|
-
failed: 1
|
|
44
|
-
scenarios:
|
|
45
|
-
- id: s01
|
|
46
|
-
name: "detects active epic from state.yaml"
|
|
47
|
-
result: PASS
|
|
48
|
-
weight: 1.0
|
|
49
|
-
- id: s02
|
|
50
|
-
name: "reads release-plan.yaml and reports next epic"
|
|
51
|
-
result: PASS
|
|
52
|
-
weight: 1.0
|
|
53
|
-
- id: s03
|
|
54
|
-
name: "handles missing state.yaml gracefully"
|
|
55
|
-
result: FAIL
|
|
56
|
-
weight: 0.5
|
|
57
|
-
failure_note: "crashed instead of suggesting state.yaml creation"
|
|
27
|
+
## Usage
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
run-benchmark <skill-name> # benchmark single skill
|
|
31
|
+
run-benchmark --all # benchmark all with definitions
|
|
32
|
+
run-benchmark <skill-name> --baseline # pin results as baseline
|
|
58
33
|
```
|
|
59
34
|
|
|
60
|
-
|
|
35
|
+
## Process
|
|
61
36
|
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
37
|
+
1. **Locate definition** — Read `specs/benchmarks/<skill>.yaml`. If absent, stop with message.
|
|
38
|
+
|
|
39
|
+
2. **Partition scenarios** — Split by `split` field (`train` → iteration, `validation` → authoritative, default: `validation`).
|
|
40
|
+
|
|
41
|
+
3. **Run each scenario (N-run delta)** — For each scenario, run grader N times (default 3, configurable via `runs:`):
|
|
42
|
+
- **Without skill:** Agent with only CLAUDE.md/CONVENTIONS.md
|
|
43
|
+
- **With skill:** Agent with the skill under test active
|
|
44
|
+
- Code grader: `bash -c <command>`, exit 0 → PASS. Timeout: 15s.
|
|
45
|
+
- Rubric grader: yes/no per criterion, ≥ 80% yes → PASS.
|
|
46
|
+
- Record: `{scenario_id: {with: [P/F,...], without: [P/F,...]}}`
|
|
47
|
+
|
|
48
|
+
4. **Calculate scores** — Per split (train, validation) and mode (with, without):
|
|
49
|
+
- `pass@k = sum(weight × pass_rate) / sum(weights)` where `pass_rate = passes/runs`
|
|
50
|
+
- `Δ = pass@k_with − pass@k_without` — causal contribution
|
|
51
|
+
- Round to 2 decimal places
|
|
52
|
+
|
|
53
|
+
5. **Write benchmark.json** to `specs/benchmarks/reports/benchmark-<skill>.json`:
|
|
54
|
+
```json
|
|
55
|
+
{"skill":"survey-context","run_date":"2026-06-22","runs_per_scenario":3,"train":{"with_skill":0.92,"without_skill":0.67,"delta":0.25,"scenarios":["s01","s02"]},"validation":{"with_skill":0.83,"without_skill":0.60,"delta":0.23,"scenarios":["s03","s04","s05"]}}
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
6. **Write YAML report** to `specs/benchmarks/reports/BENCHMARK-<skill>-<YYYY-MM-DD>.yaml`:
|
|
59
|
+
```yaml
|
|
60
|
+
skill: survey-context
|
|
61
|
+
run_date: "2026-06-22"
|
|
62
|
+
runs_per_scenario: 3
|
|
63
|
+
train:
|
|
64
|
+
pass_at_k_with: 0.92
|
|
65
|
+
pass_at_k_without: 0.67
|
|
66
|
+
delta: 0.25
|
|
67
|
+
validation:
|
|
68
|
+
pass_at_k_with: 0.83
|
|
69
|
+
pass_at_k_without: 0.60
|
|
70
|
+
delta: 0.23
|
|
71
|
+
scenarios:
|
|
72
|
+
- id: s01
|
|
73
|
+
split: train
|
|
74
|
+
with_pass_rate: 1.0
|
|
75
|
+
without_pass_rate: 0.67
|
|
76
|
+
delta: 0.33
|
|
77
|
+
weight: 1.0
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
7. **Baseline** (`--baseline`) — Copy to `BASELINE-<skill>.yaml` + `baseline-<skill>.json`.
|
|
81
|
+
|
|
82
|
+
8. **Compare to baseline** — `IMPROVED: Δ 0.17 → 0.25` / `REGRESSION: Δ 0.25 → 0.17 — do NOT ship` / `STABLE`.
|
|
83
|
+
|
|
84
|
+
9. **Delta threshold gate** — Validation Δ < 0.0 blocks release. Δ < 0.05 warns (marginal). Min meaningful threshold: 0.05.
|
|
@@ -23,6 +23,10 @@ description: > AI-powered security analysis of code changes — traces data flow
|
|
|
23
23
|
|
|
24
24
|
Covered: SQLi, XSS, SSRF, command injection, auth bypass, unsafe deserialization, path traversal, IDOR, crypto flaws, secrets exposure, template injection, NoSQLi
|
|
25
25
|
|
|
26
|
+
## BCP Plus Integration
|
|
27
|
+
|
|
28
|
+
This skill maps to **BCP Plus dimension 12 (Security & Compliance)**. When BCP Plus sizing is active, the threat model categories above correspond to sub-elements within dimension 12. The NFR Gate rule applies: standard-expectation items (e.g., "use HTTPS", "hash passwords") score 0 with a one-line rationale; only above-standard security requirements contribute to the dimension 12 count. See `docs/references/bcp-plus.md` for the full 13-dimension framework and NFR Gate pattern.
|
|
29
|
+
|
|
26
30
|
## Integration points
|
|
27
31
|
|
|
28
32
|
| Skill | Touchpoint |
|
|
@@ -18,6 +18,21 @@ Idempotent prep so BUILD phase commands succeed on first run.
|
|
|
18
18
|
5. Run smoke: lint + one fast test or `--version` on key tools.
|
|
19
19
|
6. Record versions in `specs/state.yaml` under Environment.
|
|
20
20
|
|
|
21
|
+
## BCP Plus Counter (optional)
|
|
22
|
+
|
|
23
|
+
The `big-counter` tool is an optional dependency for BCP Plus 13-dimension story sizing:
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
# Install from PyPI (recommended)
|
|
27
|
+
pip install big-counter
|
|
28
|
+
|
|
29
|
+
# Or from npm
|
|
30
|
+
npm install -g big-counter
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Verify the install: `big-counter --version`
|
|
34
|
+
Skip if BCP Plus sizing is not needed for this project.
|
|
35
|
+
|
|
21
36
|
## Verify
|
|
22
37
|
|
|
23
38
|
→ verify: `test -f CLAUDE.md && grep -q 'Test' CLAUDE.md && echo "OK: CLAUDE.md has test table" || echo "MISSING: test commands"`
|
|
@@ -90,3 +90,7 @@ Place setup scripts in `scripts/setup.sh` (or language-appropriate equivalent).
|
|
|
90
90
|
- [ ] Run `scripts/setup.sh` twice — second run should produce no errors
|
|
91
91
|
- [ ] Health check command returns success
|
|
92
92
|
- [ ] No sensitive data in log output
|
|
93
|
+
|
|
94
|
+
## BCP Plus Integration
|
|
95
|
+
|
|
96
|
+
This skill maps to **BCP Plus dimension 11 (Quality Attributes)** — specifically performance, reliability, and availability instrumentation. Structured logging, health checks, and idempotent setup scripts are observable manifestations of quality-attribute requirements. The NFR Gate rule applies: standard-expectation items (e.g., "the app should log errors") score 0 with a one-line rationale; only above-standard quality requirements (e.g., p99 latency tracking, SLO-based alerting) contribute to the dimension 11 count. See `docs/references/bcp-plus.md` for the full 13-dimension framework and NFR Gate pattern.
|
|
@@ -35,7 +35,8 @@ Orchestrates the **build** flow for a single epic: survey → plan tasks → kic
|
|
|
35
35
|
2. **Step 0 — Threat Model:** Run `security-review` against the epic's scope (read from the epic capsule). Output `specs/security/epics/<epic-id>/THREAT_MODEL.md` with surface area, vulnerability categories, risk level, and mitigation guidance.
|
|
36
36
|
3. **Assess Impact (Step 2):** Before writing tasks, run `assess-impact --lightweight` on the proposed change. If the risk score exceeds 7, gate — require a `grill-me` session. Write the impact report to `specs/IMPACT-<epic>-<story>.md`. For net-new code with no existing dependents, skip.
|
|
37
37
|
4. **BCP Tracking (Step 2):** After `plan-work` completes, read the `bcps:` count (Business Complexity Points story size) from the epic capsule and carry it into `state.yaml` as `epic_cycle.story_bcps = N`.
|
|
38
|
-
5.
|
|
38
|
+
5. **BCP Plus (Step 2, optional):** When the epic capsule carries a `bcp_plus_breakdown` (13-dimension sizing), also carry it into `state.yaml` as `epic_cycle.bcp_plus`. The breakdown maps each of the 13 dimensions to an integer count. NFR dimensions (11–13) that are N/A per the NFR Gate may be set to 0.
|
|
39
|
+
6. If `epic_cycle.step` missing, set to `1`.
|
|
39
40
|
6. Run **only the current step** (resume mode) unless user asked for full auto-run.
|
|
40
41
|
7. After step verify passes, increment `epic_cycle.step` in `state.yaml` (or `bash scripts/bp-yaml-set.sh` if available).
|
|
41
42
|
8. On story complete, set `execution-status.yaml` story key to `done`; run `bash scripts/sync-status-from-epics.sh`.
|
|
@@ -103,6 +103,10 @@ If this cycle modified files in `.github/workflows/`, run the CI dry-run procedu
|
|
|
103
103
|
Gate: READY -> next: verify-work
|
|
104
104
|
Writes: state.yaml handoff.next_skill = verify-work
|
|
105
105
|
|
|
106
|
+
## BCP Plus Integration
|
|
107
|
+
|
|
108
|
+
At story completion, if the story was sized with BCP Plus (13-dimension breakdown), log the `bcp_plus.total` alongside the standard `bcps:` count in the story's tasks.yaml. The breakdown is available from the epic capsule's `bcp_plus_breakdown` field. See `docs/references/bcp-plus.md` for the full methodology and NFR Gate pattern.
|
|
109
|
+
|
|
106
110
|
---
|
|
107
111
|
|
|
108
112
|
# Develop TDD — Reference
|
|
@@ -119,3 +119,8 @@ A numbered list of RED-GREEN cycles:
|
|
|
119
119
|
</diagnosis-template>
|
|
120
120
|
|
|
121
121
|
After writing the bug file, print a one-line summary of the root cause and suggest running `kickoff-branch` next to create a fix branch.
|
|
122
|
+
|
|
123
|
+
## References
|
|
124
|
+
|
|
125
|
+
- [Feathers' Seams & Characterization Tests](../../docs/references/feathers.md) — seam types, characterization tests, and the legacy code change algorithm
|
|
126
|
+
- [Fowler's Code Smells](../../docs/references/fowler.md) — identifying structural problems by smell before investigating
|
|
@@ -78,4 +78,9 @@ Any further notes about the refactor.
|
|
|
78
78
|
|
|
79
79
|
After writing `specs/REFACTOR_LATEST.md`, suggest running `kickoff-branch` next to create a refactor branch.
|
|
80
80
|
|
|
81
|
+
## References
|
|
82
|
+
|
|
83
|
+
- [Fowler's Refactoring Catalog](../../docs/references/fowler.md) — canonical refactoring vocabulary and code-smell taxonomy
|
|
84
|
+
- [Beck's Tidy First?](../../docs/references/kent-beck.md) — structural change before behavioral change
|
|
85
|
+
|
|
81
86
|
## Verify
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: run-benchmark
|
|
3
|
-
description: "Run skill quality benchmarks from specs/benchmarks/ definitions
|
|
3
|
+
description: "Run skill quality benchmarks from specs/benchmarks/ definitions — N-run with/without-skill delta grading, train/validation split, pass@k + benchmark.json reports. Use before and after evolve-skill to prove quality changes are improvements, not regressions."
|
|
4
4
|
model: haiku
|
|
5
5
|
---
|
|
6
6
|
|
|
@@ -9,59 +9,78 @@ model: haiku
|
|
|
9
9
|
|
|
10
10
|
> **HARD GATE** — Do NOT use benchmark scores to declare a skill "good" or "bad" in isolation. Benchmarks measure relative quality vs. a baseline — they catch regressions, they do not certify correctness.
|
|
11
11
|
|
|
12
|
-
Reads benchmark definitions from `specs/benchmarks/`, executes each scenario's grader, and writes a structured `pass@k` report that `evolve-skill` consumes.
|
|
12
|
+
Reads benchmark definitions from `specs/benchmarks/`, executes each scenario's grader with and without the skill loaded, and writes a structured `pass@k` report with delta grading that `evolve-skill` consumes.
|
|
13
13
|
|
|
14
|
-
##
|
|
14
|
+
## With/Without-Skill Delta Grading
|
|
15
15
|
|
|
16
|
-
|
|
17
|
-
# Benchmark a single skill
|
|
18
|
-
run-benchmark <skill-name>
|
|
16
|
+
Every scenario runs N times (default 3) in two modes: **with** the skill loaded and **without** (bare agent with only CLAUDE.md). The delta `Δ = pass@k_with − pass@k_without` isolates the skill's causal contribution. A negative delta is a regression flag.
|
|
19
17
|
|
|
20
|
-
|
|
21
|
-
run-benchmark --all
|
|
18
|
+
## Train/Validation Split
|
|
22
19
|
|
|
23
|
-
|
|
24
|
-
run-benchmark <skill-name> --baseline
|
|
25
|
-
```
|
|
20
|
+
Benchmark definitions partition scenarios into two sets:
|
|
26
21
|
|
|
27
|
-
|
|
22
|
+
| Set | Tag | Purpose |
|
|
23
|
+
|-----|-----|---------|
|
|
24
|
+
| **Train** | `split: train` | Development scenarios — used while iterating. Hitting 100% on train is expected. |
|
|
25
|
+
| **Validation** | `split: validation` | Held-out scenarios — the real quality signal. Overfitting train while validation stagnates is a design smell. |
|
|
26
|
+
|
|
27
|
+
`pass@k` is reported separately for train and validation. Validation score is authoritative; train score is iteration guidance only.
|
|
28
28
|
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
3. **Calculate pass@k** — `pass@k = sum(weight of PASS scenarios) / sum(all weights)`. Round to 2 decimal places.
|
|
36
|
-
|
|
37
|
-
4. **Write report** to `specs/benchmarks/reports/BENCHMARK-<skill>-<YYYY-MM-DD>.yaml`:
|
|
38
|
-
|
|
39
|
-
```yaml
|
|
40
|
-
skill: survey-context
|
|
41
|
-
run_date: "2026-06-22"
|
|
42
|
-
pass_at_k: 0.83
|
|
43
|
-
total_scenarios: 3
|
|
44
|
-
passed: 2
|
|
45
|
-
failed: 1
|
|
46
|
-
scenarios:
|
|
47
|
-
- id: s01
|
|
48
|
-
name: "detects active epic from state.yaml"
|
|
49
|
-
result: PASS
|
|
50
|
-
weight: 1.0
|
|
51
|
-
- id: s02
|
|
52
|
-
name: "reads release-plan.yaml and reports next epic"
|
|
53
|
-
result: PASS
|
|
54
|
-
weight: 1.0
|
|
55
|
-
- id: s03
|
|
56
|
-
name: "handles missing state.yaml gracefully"
|
|
57
|
-
result: FAIL
|
|
58
|
-
weight: 0.5
|
|
59
|
-
failure_note: "crashed instead of suggesting state.yaml creation"
|
|
29
|
+
## Usage
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
run-benchmark <skill-name> # benchmark single skill
|
|
33
|
+
run-benchmark --all # benchmark all with definitions
|
|
34
|
+
run-benchmark <skill-name> --baseline # pin results as baseline
|
|
60
35
|
```
|
|
61
36
|
|
|
62
|
-
|
|
37
|
+
## Process
|
|
63
38
|
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
39
|
+
1. **Locate definition** — Read `specs/benchmarks/<skill>.yaml`. If absent, stop with message.
|
|
40
|
+
|
|
41
|
+
2. **Partition scenarios** — Split by `split` field (`train` → iteration, `validation` → authoritative, default: `validation`).
|
|
42
|
+
|
|
43
|
+
3. **Run each scenario (N-run delta)** — For each scenario, run grader N times (default 3, configurable via `runs:`):
|
|
44
|
+
- **Without skill:** Agent with only CLAUDE.md/CONVENTIONS.md
|
|
45
|
+
- **With skill:** Agent with the skill under test active
|
|
46
|
+
- Code grader: `bash -c <command>`, exit 0 → PASS. Timeout: 15s.
|
|
47
|
+
- Rubric grader: yes/no per criterion, ≥ 80% yes → PASS.
|
|
48
|
+
- Record: `{scenario_id: {with: [P/F,...], without: [P/F,...]}}`
|
|
49
|
+
|
|
50
|
+
4. **Calculate scores** — Per split (train, validation) and mode (with, without):
|
|
51
|
+
- `pass@k = sum(weight × pass_rate) / sum(weights)` where `pass_rate = passes/runs`
|
|
52
|
+
- `Δ = pass@k_with − pass@k_without` — causal contribution
|
|
53
|
+
- Round to 2 decimal places
|
|
54
|
+
|
|
55
|
+
5. **Write benchmark.json** to `specs/benchmarks/reports/benchmark-<skill>.json`:
|
|
56
|
+
```json
|
|
57
|
+
{"skill":"survey-context","run_date":"2026-06-22","runs_per_scenario":3,"train":{"with_skill":0.92,"without_skill":0.67,"delta":0.25,"scenarios":["s01","s02"]},"validation":{"with_skill":0.83,"without_skill":0.60,"delta":0.23,"scenarios":["s03","s04","s05"]}}
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
6. **Write YAML report** to `specs/benchmarks/reports/BENCHMARK-<skill>-<YYYY-MM-DD>.yaml`:
|
|
61
|
+
```yaml
|
|
62
|
+
skill: survey-context
|
|
63
|
+
run_date: "2026-06-22"
|
|
64
|
+
runs_per_scenario: 3
|
|
65
|
+
train:
|
|
66
|
+
pass_at_k_with: 0.92
|
|
67
|
+
pass_at_k_without: 0.67
|
|
68
|
+
delta: 0.25
|
|
69
|
+
validation:
|
|
70
|
+
pass_at_k_with: 0.83
|
|
71
|
+
pass_at_k_without: 0.60
|
|
72
|
+
delta: 0.23
|
|
73
|
+
scenarios:
|
|
74
|
+
- id: s01
|
|
75
|
+
split: train
|
|
76
|
+
with_pass_rate: 1.0
|
|
77
|
+
without_pass_rate: 0.67
|
|
78
|
+
delta: 0.33
|
|
79
|
+
weight: 1.0
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
7. **Baseline** (`--baseline`) — Copy to `BASELINE-<skill>.yaml` + `baseline-<skill>.json`.
|
|
83
|
+
|
|
84
|
+
8. **Compare to baseline** — `IMPROVED: Δ 0.17 → 0.25` / `REGRESSION: Δ 0.25 → 0.17 — do NOT ship` / `STABLE`.
|
|
85
|
+
|
|
86
|
+
9. **Delta threshold gate** — Validation Δ < 0.0 blocks release. Δ < 0.05 warns (marginal). Min meaningful threshold: 0.05.
|
|
@@ -25,6 +25,10 @@ model: sonnet
|
|
|
25
25
|
|
|
26
26
|
Covered: SQLi, XSS, SSRF, command injection, auth bypass, unsafe deserialization, path traversal, IDOR, crypto flaws, secrets exposure, template injection, NoSQLi
|
|
27
27
|
|
|
28
|
+
## BCP Plus Integration
|
|
29
|
+
|
|
30
|
+
This skill maps to **BCP Plus dimension 12 (Security & Compliance)**. When BCP Plus sizing is active, the threat model categories above correspond to sub-elements within dimension 12. The NFR Gate rule applies: standard-expectation items (e.g., "use HTTPS", "hash passwords") score 0 with a one-line rationale; only above-standard security requirements contribute to the dimension 12 count. See `docs/references/bcp-plus.md` for the full 13-dimension framework and NFR Gate pattern.
|
|
31
|
+
|
|
28
32
|
## Integration points
|
|
29
33
|
|
|
30
34
|
| Skill | Touchpoint |
|
|
@@ -20,6 +20,21 @@ Idempotent prep so BUILD phase commands succeed on first run.
|
|
|
20
20
|
5. Run smoke: lint + one fast test or `--version` on key tools.
|
|
21
21
|
6. Record versions in `specs/state.yaml` under Environment.
|
|
22
22
|
|
|
23
|
+
## BCP Plus Counter (optional)
|
|
24
|
+
|
|
25
|
+
The `big-counter` tool is an optional dependency for BCP Plus 13-dimension story sizing:
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
# Install from PyPI (recommended)
|
|
29
|
+
pip install big-counter
|
|
30
|
+
|
|
31
|
+
# Or from npm
|
|
32
|
+
npm install -g big-counter
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Verify the install: `big-counter --version`
|
|
36
|
+
Skip if BCP Plus sizing is not needed for this project.
|
|
37
|
+
|
|
23
38
|
## Verify
|
|
24
39
|
|
|
25
40
|
→ verify: `test -f CLAUDE.md && grep -q 'Test' CLAUDE.md && echo "OK: CLAUDE.md has test table" || echo "MISSING: test commands"`
|
|
@@ -92,3 +92,7 @@ Place setup scripts in `scripts/setup.sh` (or language-appropriate equivalent).
|
|
|
92
92
|
- [ ] Run `scripts/setup.sh` twice — second run should produce no errors
|
|
93
93
|
- [ ] Health check command returns success
|
|
94
94
|
- [ ] No sensitive data in log output
|
|
95
|
+
|
|
96
|
+
## BCP Plus Integration
|
|
97
|
+
|
|
98
|
+
This skill maps to **BCP Plus dimension 11 (Quality Attributes)** — specifically performance, reliability, and availability instrumentation. Structured logging, health checks, and idempotent setup scripts are observable manifestations of quality-attribute requirements. The NFR Gate rule applies: standard-expectation items (e.g., "the app should log errors") score 0 with a one-line rationale; only above-standard quality requirements (e.g., p99 latency tracking, SLO-based alerting) contribute to the dimension 11 count. See `docs/references/bcp-plus.md` for the full 13-dimension framework and NFR Gate pattern.
|
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,28 @@
|
|
|
1
|
+
## [2.64.1](https://github.com/danielvm-git/bigpowers/compare/v2.64.0...v2.64.1) (2026-07-05)
|
|
2
|
+
|
|
3
|
+
|
|
4
|
+
### Bug Fixes
|
|
5
|
+
|
|
6
|
+
* **ci:** bump docs-site node to 22 for Astro 7.x compatibility ([6487a32](https://github.com/danielvm-git/bigpowers/commit/6487a3245711e9a0e05b1a36017c9cfa111b4879))
|
|
7
|
+
* **ci:** bump docs-site node to 22 for Astro 7.x compatibility ([b2df6b8](https://github.com/danielvm-git/bigpowers/commit/b2df6b87ecd594dcf3df037c93a1d671da2d13d6))
|
|
8
|
+
|
|
9
|
+
# [2.64.0](https://github.com/danielvm-git/bigpowers/compare/v2.63.0...v2.64.0) (2026-07-05)
|
|
10
|
+
|
|
11
|
+
|
|
12
|
+
### Features
|
|
13
|
+
|
|
14
|
+
* **e33:** Docs Website — fourth generated artifact target ([ff7f3ec](https://github.com/danielvm-git/bigpowers/commit/ff7f3ec6eab7c4b30a9676888468af5c9c9dd196))
|
|
15
|
+
* **e35:** missing historical references — 10 stories, 8 new ref docs ([ffba59f](https://github.com/danielvm-git/bigpowers/commit/ffba59f3ebbe18a72385a3f325a5856a8a53da9e))
|
|
16
|
+
* **e37s01:** BCP Plus counter integration — setup-environment + bcp-plus reference ([f2b4bdf](https://github.com/danielvm-git/bigpowers/commit/f2b4bdf86b6256089b593207aec37480e789edb7))
|
|
17
|
+
* **e37s03:** NFR Gate integration — security-review and wire-observability sizing ([de47871](https://github.com/danielvm-git/bigpowers/commit/de478716f22e1b576a4b936103144aac39e921a4))
|
|
18
|
+
* **e37s04:** Build-epic integration — BCP Plus in story sizing workflow ([a961e6b](https://github.com/danielvm-git/bigpowers/commit/a961e6be3c7f6a984654b38c4a867423ac6c11db))
|
|
19
|
+
* **e43:** showcase repo — full worked example + sync-skills regeneration ([a872ca8](https://github.com/danielvm-git/bigpowers/commit/a872ca82745c6e7b983ef45f42f3251d3df0d680))
|
|
20
|
+
* **e43:** showcase repo — full worked example of bigpowers methodology ([e32373b](https://github.com/danielvm-git/bigpowers/commit/e32373bff09cd0bf01a06406a95fccff8ad54f47)), closes [#dependent](https://github.com/danielvm-git/bigpowers/issues/dependent)
|
|
21
|
+
* **e47:** cross-tool skill distribution — pi install, seed wiring, docs, verify harness ([0109c2f](https://github.com/danielvm-git/bigpowers/commit/0109c2fd87e2c2ef783e335a7e3d10e268ba5162))
|
|
22
|
+
* **e48s01:** benchmark schema v2 — split/runs fields, story tags ([47bc1ef](https://github.com/danielvm-git/bigpowers/commit/47bc1ef92e6c34b01dc199f22b6edf357f5f0120))
|
|
23
|
+
* **e48s01:** run-benchmark — delta grading + train/validation split ([d2d1fe8](https://github.com/danielvm-git/bigpowers/commit/d2d1fe84ec923f34454b7bdac40205667f18760f))
|
|
24
|
+
* **e48s01:** run-benchmark — with/without-skill delta grading + train/validation split ([651d0d8](https://github.com/danielvm-git/bigpowers/commit/651d0d8b2c44477fa3be08162a5e0ecde2db2258))
|
|
25
|
+
|
|
1
26
|
# [2.63.0](https://github.com/danielvm-git/bigpowers/compare/v2.62.0...v2.63.0) (2026-07-05)
|
|
2
27
|
|
|
3
28
|
|
package/CLAUDE.md
CHANGED
|
@@ -35,6 +35,8 @@ npm run compliance && bash scripts/run-golden-suite.sh
|
|
|
35
35
|
|
|
36
36
|
If any gate fails, fix before merging. Run `--baseline` after any intentional increase in skill count or structure.
|
|
37
37
|
|
|
38
|
+
**BCP Plus:** For stories sized with the 13-dimension BCP Plus methodology, confirm the `bcp_plus_breakdown` is present in the epic YAML and carried into `state.yaml` as `epic_cycle.bcp_plus`. See `docs/references/bcp-plus.md` for the NFR Gate pattern.
|
|
39
|
+
|
|
38
40
|
## Architecture
|
|
39
41
|
|
|
40
42
|
Collection of verb-noun skills under `skills/`, each with a SKILL.md source file and supporting documentation. Runtime specs live in `specs/state.yaml`, `specs/release-plan.yaml`, and `specs/execution-status.yaml`; intent in `specs/product/`; epic shards in `specs/epics/`. The sync-skills.sh script auto-generates artifacts for Cursor (.cursor/rules) and Gemini CLI (.gemini/extensions/bigpowers/) from SKILL.md sources. All planning output goes to specs/ at the project root.
|
|
@@ -46,10 +48,12 @@ Collection of verb-noun skills under `skills/`, each with a SKILL.md source file
|
|
|
46
48
|
- All planning/spec output goes to specs/ at project root
|
|
47
49
|
- Artifacts in .cursor/rules and .gemini/ are auto-generated; edit SKILL.md, not artifacts
|
|
48
50
|
- Run sync-skills.sh after any SKILL.md changes to regenerate artifacts
|
|
51
|
+
- Website content in website/src/content/docs/ is auto-generated by prebuild; edit repo sources, not site files
|
|
49
52
|
|
|
50
53
|
## Never
|
|
51
54
|
|
|
52
55
|
- Never edit .cursor/rules or .gemini/extensions/ directly — these are generated files
|
|
56
|
+
- Never edit website/src/content/docs/ directly — these are generated files; website/ is the fourth generated artifact target (alongside .cursor/, .gemini/, .pi/)
|
|
53
57
|
- Never create a skill without a SKILL.md file and proper verb-noun naming
|
|
54
58
|
- Never push changes without running sync-skills.sh first
|
|
55
59
|
|
package/CONVENTIONS.md
CHANGED
|
@@ -140,6 +140,16 @@ When planning closes, copy to `specs/product/snapshots/release-<version>/` (`rel
|
|
|
140
140
|
|
|
141
141
|
Validate YAML layout: `bash scripts/validate-specs-yaml.sh`. Patch runtime keys: `bash scripts/bp-yaml-set.sh specs/state.yaml git.branch feat/foo`.
|
|
142
142
|
|
|
143
|
+
### Generated artifact targets
|
|
144
|
+
|
|
145
|
+
Skill content is edited in `skills/*/SKILL.md` and auto-propagated to these targets by `bash scripts/sync-skills.sh`:
|
|
146
|
+
- `.cursor/rules/` — Cursor IDE rules
|
|
147
|
+
- `.gemini/extensions/bigpowers/` — Gemini CLI extensions
|
|
148
|
+
- `.pi/skills/` + `.pi/prompts/` — pi agent
|
|
149
|
+
- `website/src/content/docs/` — Docs website (Astro Starlight, deployed to GitHub Pages)
|
|
150
|
+
|
|
151
|
+
**Never edit any of these targets directly.** Edit the SKILL.md sources and run rebuild scripts (`bash scripts/sync-skills.sh`, `npm run site:build`).
|
|
152
|
+
|
|
143
153
|
### Legacy paths (migrate away)
|
|
144
154
|
|
|
145
155
|
| Old | New |
|
package/README.md
CHANGED
|
@@ -12,6 +12,12 @@ It is not a random collection of best practices. It is a chronological layer cak
|
|
|
12
12
|
|
|
13
13
|
Published on npm: [bigpowers](https://www.npmjs.com/package/bigpowers). The skill count in the badge above is stamped automatically by `sync-skills.sh`; the canonical catalog is [`SKILL-INDEX.md`](SKILL-INDEX.md).
|
|
14
14
|
|
|
15
|
+
Docs: [bigpowers docs site](https://danielvm-git.github.io/bigpowers/) — searchable, Google-discoverable reference for all skills, guides, and ADRs.
|
|
16
|
+
|
|
17
|
+
**This methodology publishes its own evidence — see the [live receipts page](https://danielvm-git.github.io/bigpowers/receipts/).**
|
|
18
|
+
|
|
19
|
+
**See it working:** [bigpowers-showcase](https://github.com/danielvm-git/bigpowers-showcase) — a real URL shortener (CLI + SQLite) built from scratch with the full spec trail committed from day one.
|
|
20
|
+
|
|
15
21
|
---
|
|
16
22
|
|
|
17
23
|
## 🗺 How to Read This README
|
package/SKILL-INDEX.md
CHANGED
|
@@ -91,7 +91,7 @@
|
|
|
91
91
|
| 63 | Sustain | `migrate-spec` | Detect GSD, spec-kit, or BMAD spec artifacts and transform them into bigpowers Y | ✅ Active |
|
|
92
92
|
| 64 | Sustain | `organize-workspace` | Scans the active workspace for disposable artifacts—logs, caches, stale build | ✅ Active |
|
|
93
93
|
| 65 | Sustain | `reset-baseline` | Restore the project to a known clean state between agent runs or experiments. Us | ✅ Active |
|
|
94
|
-
| 66 | Sustain | `run-benchmark` | Run skill quality benchmarks from specs/benchmarks/ definitions
|
|
94
|
+
| 66 | Sustain | `run-benchmark` | Run skill quality benchmarks from specs/benchmarks/ definitions — N-run with/w | ✅ Active |
|
|
95
95
|
| 67 | Sustain | `session-state` | Track implementation decisions and progress in specs/state.yaml to prevent conte | ✅ Active |
|
|
96
96
|
| 68 | Sustain | `simulate-agents` | Run Mock User and Auditor agents against a feature in fresh contexts before huma | ✅ Active |
|
|
97
97
|
| 69 | Sustain | `stocktake-skills` | Sequential subagent batch audit of the bigpowers skill catalog — Quick Scan (c | ✅ Active |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "bigpowers",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.64.1",
|
|
4
4
|
"description": "73 agent skills synthesizing 17 years of software engineering discipline into a prescriptive methodology for solo developers",
|
|
5
5
|
"main": "index.js",
|
|
6
6
|
"scripts": {
|
|
@@ -12,7 +12,10 @@
|
|
|
12
12
|
"version": "bash scripts/sync-skills.sh && git add .gemini/extensions/bigpowers/gemini-extension.json specs/SKILL-SEARCH-INDEX_LATEST.md",
|
|
13
13
|
"release": "semantic-release",
|
|
14
14
|
"dashboard": "node dashboard/bin/dashboard.js",
|
|
15
|
-
"dashboard:web": "node dashboard/bin/dashboard.js --web"
|
|
15
|
+
"dashboard:web": "node dashboard/bin/dashboard.js --web",
|
|
16
|
+
"site:dev": "cd website && npm run dev",
|
|
17
|
+
"site:build": "cd website && npm run build",
|
|
18
|
+
"site:preview": "cd website && npm run preview"
|
|
16
19
|
},
|
|
17
20
|
"keywords": [
|
|
18
21
|
"skills",
|