@polderlabs/bizar 10.6.0 → 10.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bizar-dash/tests/memory-protocol-drift.test.mjs +18 -11
- package/cli/__tests__/feature-list-bridge.test.mjs +42 -42
- package/cli/agent-browser-up.sh +35 -35
- package/cli/agent-browser-update.mjs +42 -42
- package/cli/agent-browser-update.test.mjs +7 -7
- package/cli/audit.mjs +3 -3
- package/cli/bin.mjs +10 -10
- package/cli/bootstrap.mjs +1 -1
- package/cli/cli-commands-validation.test.mjs +2 -2
- package/cli/commands/claim.mjs +3 -3
- package/cli/commands/claude-cmd.mjs +3 -3
- package/cli/commands/eval.mjs +6 -6
- package/cli/commands/util.mjs +18 -18
- package/cli/commands/validate.mjs +10 -10
- package/cli/init.mjs +1 -1
- package/cli/install/postinstall.mjs +4 -4
- package/cli/plow-through.test.mjs +7 -6
- package/cli/prompts.mjs +11 -11
- package/cli/provision.mjs +4 -16
- package/config/skills/bizar/SKILL.md +1 -1
- package/config/skills/skillopt/SKILL.md +129 -0
- package/config/skills/thinking-archetypes/SKILL.md +90 -0
- package/config/skills/thinking-bayesian/SKILL.md +267 -0
- package/config/skills/thinking-bounded-rationality/SKILL.md +406 -0
- package/config/skills/thinking-circle-of-competence/SKILL.md +216 -0
- package/config/skills/thinking-cynefin/SKILL.md +70 -0
- package/config/skills/thinking-debiasing/SKILL.md +192 -0
- package/config/skills/thinking-dual-process/SKILL.md +282 -0
- package/config/skills/thinking-effectuation/SKILL.md +366 -0
- package/config/skills/thinking-feedback-loops/SKILL.md +464 -0
- package/config/skills/thinking-fermi-estimation/SKILL.md +263 -0
- package/config/skills/thinking-first-principles/SKILL.md +167 -0
- package/config/skills/thinking-five-whys-plus/SKILL.md +139 -0
- package/config/skills/thinking-inversion/SKILL.md +195 -0
- package/config/skills/thinking-jobs-to-be-done/SKILL.md +363 -0
- package/config/skills/thinking-kepner-tregoe/SKILL.md +154 -0
- package/config/skills/thinking-leverage-points/SKILL.md +390 -0
- package/config/skills/thinking-lindy-effect/SKILL.md +331 -0
- package/config/skills/thinking-map-territory/SKILL.md +111 -0
- package/config/skills/thinking-margin-of-safety/SKILL.md +330 -0
- package/config/skills/thinking-model-combination/SKILL.md +406 -0
- package/config/skills/thinking-model-router/SKILL.md +360 -0
- package/config/skills/thinking-model-selection/SKILL.md +341 -0
- package/config/skills/thinking-occams-razor/SKILL.md +129 -0
- package/config/skills/thinking-ooda/SKILL.md +127 -0
- package/config/skills/thinking-opportunity-cost/SKILL.md +360 -0
- package/config/skills/thinking-pre-mortem/SKILL.md +170 -0
- package/config/skills/thinking-probabilistic/SKILL.md +324 -0
- package/config/skills/thinking-red-team/SKILL.md +142 -0
- package/config/skills/thinking-regret-minimization/SKILL.md +335 -0
- package/config/skills/thinking-reversibility/SKILL.md +326 -0
- package/config/skills/thinking-scientific-method/SKILL.md +162 -0
- package/config/skills/thinking-second-order/SKILL.md +184 -0
- package/config/skills/thinking-socratic/SKILL.md +198 -0
- package/config/skills/thinking-steel-manning/SKILL.md +332 -0
- package/config/skills/thinking-systems/SKILL.md +238 -0
- package/config/skills/thinking-theory-of-constraints/SKILL.md +338 -0
- package/config/skills/thinking-thought-experiment/SKILL.md +354 -0
- package/config/skills/thinking-triz/SKILL.md +171 -0
- package/config/skills/thinking-via-negativa/SKILL.md +358 -0
- package/package.json +2 -3
- package/packages/sdk/dist/consensus/index.js +4 -4
- package/packages/sdk/dist/router/agent-model-registry.js +2 -2
- package/packages/sdk/dist/router/index.d.ts +3 -3
- package/packages/sdk/dist/router/index.js +4 -4
- package/packages/sdk/dist/router/q-learning-router.d.ts +2 -2
- package/packages/sdk/dist/router/q-learning-router.js +18 -18
- package/scripts/bh-full-e2e.mjs +10 -8
- package/scripts/check-agents.mjs +27 -24
- package/scripts/mirror-claude-md.sh +0 -0
- package/scripts/sync-skills-mirror.mjs +113 -0
- package/scripts/verify-thinking-skills.mjs +135 -0
- package/config/agents/_shared/AGENT_BASELINE.md +0 -155
- package/config/agents/_shared/CLINE_TOOLS.md +0 -398
- package/config/agents/_shared/SKILLS.md +0 -109
- package/config/agents/agent-browser.md +0 -90
- package/config/agents/baldr.md +0 -60
- package/config/agents/forseti.md +0 -60
- package/config/agents/frigg.md +0 -52
- package/config/agents/heimdall.md +0 -44
- package/config/agents/hermod.md +0 -64
- package/config/agents/mimir.md +0 -60
- package/config/agents/odin.md +0 -310
- package/config/agents/quick.md +0 -47
- package/config/agents/semble-search.md +0 -59
- package/config/agents/thor.md +0 -64
- package/config/agents/tyr.md +0 -67
- package/config/agents/vidarr.md +0 -65
- package/config/agents/vor.md +0 -62
- package/config/commands/audit.md +0 -5
- package/config/commands/bizar.md +0 -22
- package/config/commands/explain.md +0 -5
- package/config/commands/init.md +0 -26
- package/config/commands/learn.md +0 -5
- package/config/commands/plan.md +0 -30
- package/config/commands/plow-through.md +0 -47
- package/config/commands/pr-review.md +0 -5
- package/config/commands/setup-provider.md +0 -95
- package/config/commands/tailscale-serve.md +0 -97
- package/config/commands/team.md +0 -124
- package/config/commands/test.md +0 -60
- package/config/commands/validate.md +0 -66
- package/config/commands/visual-plan.md +0 -19
- package/config/hooks/PostToolUse +0 -66
- package/config/hooks/PreToolUse +0 -82
- package/config/hooks/README.md +0 -68
- package/config/hooks/TaskResume +0 -48
- package/config/hooks/TaskStart +0 -47
- package/config/hooks/UserPromptSubmit +0 -55
|
@@ -11,10 +11,10 @@ import { join } from 'node:path';
|
|
|
11
11
|
import { resolveClaudeDir, PATHS } from './paths.mjs';
|
|
12
12
|
|
|
13
13
|
const AGENT_FILES = [
|
|
14
|
-
'
|
|
15
|
-
'
|
|
16
|
-
'
|
|
17
|
-
'
|
|
14
|
+
'mike.md', 'janet.md', 'susan.md', 'pam.md',
|
|
15
|
+
'greg.md', 'brenda.md', 'steve.md', 'todd.md', 'brad.md',
|
|
16
|
+
'karen.md', 'carl.md', 'linda.md',
|
|
17
|
+
'oscar.md',
|
|
18
18
|
];
|
|
19
19
|
|
|
20
20
|
/**
|
|
@@ -11,10 +11,11 @@ import { fileURLToPath } from 'node:url';
|
|
|
11
11
|
|
|
12
12
|
const __dirname = dirname(fileURLToPath(import.meta.url));
|
|
13
13
|
const PROJECT_ROOT = join(__dirname, '..');
|
|
14
|
-
|
|
14
|
+
// F-107: slash commands canonical at .claude/commands/. Legacy config/commands/ removed (Cline-era).
|
|
15
|
+
const CMD_PATH = join(PROJECT_ROOT, '.claude', 'commands', 'plow-through.md');
|
|
15
16
|
|
|
16
17
|
describe('/plow-through command file', () => {
|
|
17
|
-
test('file exists at
|
|
18
|
+
test('file exists at .claude/commands/plow-through.md', () => {
|
|
18
19
|
assert.equal(existsSync(CMD_PATH), true, 'plow-through.md must exist');
|
|
19
20
|
});
|
|
20
21
|
|
|
@@ -23,10 +24,10 @@ describe('/plow-through command file', () => {
|
|
|
23
24
|
assert.match(content, /^---\ndescription:/m, 'frontmatter must have description');
|
|
24
25
|
});
|
|
25
26
|
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
27
|
+
// F-107: legacy `agent: mike` frontmatter field was removed when slash
|
|
28
|
+
// commands were migrated to Claude Code's `run_in_background: true`
|
|
29
|
+
// Agent dispatch model. Routed agents are now inferred from the
|
|
30
|
+
// command body via the thinking-route hook.
|
|
30
31
|
|
|
31
32
|
test('body has content (more than 5 lines)', () => {
|
|
32
33
|
const content = readFileSync(CMD_PATH, 'utf8');
|
package/cli/prompts.mjs
CHANGED
|
@@ -1,17 +1,17 @@
|
|
|
1
1
|
import inquirer from 'inquirer';
|
|
2
2
|
|
|
3
3
|
const AGENTS_LIST = [
|
|
4
|
-
{ name: 'Odin ᛟ — MiniMax-M3 (Router)', value: '
|
|
5
|
-
{ name: 'Frigg ᚠ — DeepSeek (Codebase Q&A, read-only)', value: '
|
|
6
|
-
{ name: 'Vör ᚡ — DeepSeek (Clarification)', value: '
|
|
7
|
-
{ name: 'Mimir ᛗ — DeepSeek (Research)', value: '
|
|
8
|
-
{ name: 'Heimdall ᚹ — DeepSeek (Simple tasks)', value: '
|
|
9
|
-
{ name: 'Hermod ᚱ — MiniMax-M2.7 (Git ops)', value: '
|
|
10
|
-
{ name: 'Thor ᚦ — MiniMax-M2.7 (Mid impl.)', value: '
|
|
11
|
-
{ name: 'Baldr ᛒ — MiniMax-M2.7 (Design)', value: '
|
|
12
|
-
{ name: 'Tyr ᛏ — MiniMax-M3 (Complex impl.)', value: '
|
|
13
|
-
{ name: 'Vidarr ᛉ — MiniMax-M3 (Last resort)', value: '
|
|
14
|
-
{ name: 'Forseti ᚨ — MiniMax-M3 (Plan audit)', value: '
|
|
4
|
+
{ name: 'Odin ᛟ — MiniMax-M3 (Router)', value: 'mike.md', checked: true },
|
|
5
|
+
{ name: 'Frigg ᚠ — DeepSeek (Codebase Q&A, read-only)', value: 'susan.md', checked: true },
|
|
6
|
+
{ name: 'Vör ᚡ — DeepSeek (Clarification)', value: 'janet.md', checked: true },
|
|
7
|
+
{ name: 'Mimir ᛗ — DeepSeek (Research)', value: 'greg.md', checked: true },
|
|
8
|
+
{ name: 'Heimdall ᚹ — DeepSeek (Simple tasks)', value: 'brenda.md', checked: true },
|
|
9
|
+
{ name: 'Hermod ᚱ — MiniMax-M2.7 (Git ops)', value: 'steve.md', checked: true },
|
|
10
|
+
{ name: 'Thor ᚦ — MiniMax-M2.7 (Mid impl.)', value: 'todd.md', checked: true },
|
|
11
|
+
{ name: 'Baldr ᛒ — MiniMax-M2.7 (Design)', value: 'brad.md', checked: true },
|
|
12
|
+
{ name: 'Tyr ᛏ — MiniMax-M3 (Complex impl.)', value: 'karen.md', checked: true },
|
|
13
|
+
{ name: 'Vidarr ᛉ — MiniMax-M3 (Last resort)', value: 'carl.md', checked: false },
|
|
14
|
+
{ name: 'Forseti ᚨ — MiniMax-M3 (Plan audit)', value: 'linda.md', checked: true },
|
|
15
15
|
];
|
|
16
16
|
|
|
17
17
|
export async function promptComponents() {
|
package/cli/provision.mjs
CHANGED
|
@@ -430,21 +430,10 @@ function syncDir(srcDir, destDir, opts = {}) {
|
|
|
430
430
|
return { copied, skipped };
|
|
431
431
|
}
|
|
432
432
|
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
437
|
-
if (dryRun) return { ok: true, message: `[dry-run] would sync ${src} → ${dest}` };
|
|
438
|
-
ensureDir(dest); ensureDir(join(dest, '_shared'));
|
|
439
|
-
let { copied, skipped } = syncDir(src, dest, { filter: n => n.endsWith('.md') });
|
|
440
|
-
const sharedSrc = join(src, '_shared');
|
|
441
|
-
if (existsSync(sharedSrc)) {
|
|
442
|
-
for (const entry of readdirSync(sharedSrc, { withFileTypes: true })) {
|
|
443
|
-
if (entry.isFile()) { copyFileSync(join(sharedSrc, entry.name), join(dest, '_shared', entry.name)); copied++; }
|
|
444
|
-
}
|
|
445
|
-
}
|
|
446
|
-
return { ok: true, message: `${copied} agent(s) synced (${skipped} kept)`, copied, skipped };
|
|
447
|
-
}
|
|
433
|
+
// F-107: syncAgentFiles removed. Agent definitions live at
|
|
434
|
+
// .claude/agents/ (Claude Code canonical) — they were never sourced
|
|
435
|
+
// from config/agents/ in Claude Code era, and the legacy source dir
|
|
436
|
+
// was deleted.
|
|
448
437
|
|
|
449
438
|
export async function syncSkillFiles({ dryRun = false } = {}) {
|
|
450
439
|
const src = join(REPO_ROOT, 'config', 'skills');
|
|
@@ -704,7 +693,6 @@ export async function runProvision(opts = {}) {
|
|
|
704
693
|
return r;
|
|
705
694
|
};
|
|
706
695
|
|
|
707
|
-
await runStep('Syncing agents', () => syncAgentFiles({ dryRun, force }));
|
|
708
696
|
await runStep('Syncing skills', () => syncSkillFiles({ dryRun, force }));
|
|
709
697
|
await runStep('Syncing commands', () => syncCommandFiles({ dryRun, force }));
|
|
710
698
|
await runStep('Syncing rules', () => syncRulesFiles({ dryRun, force }));
|
|
@@ -253,7 +253,7 @@ Single-key mode is unchanged — if only `MINIMAX_API_KEY` is set (or no key rot
|
|
|
253
253
|
|
|
254
254
|
## Verbose Agent Baseline Reference
|
|
255
255
|
|
|
256
|
-
> The 12 always-on rules in
|
|
256
|
+
> The 12 always-on rules in `.claude/agents/_shared/AGENT_BASELINE.md`
|
|
257
257
|
> are auto-loaded into every agent session at startup. The full prose
|
|
258
258
|
> for each rule lives below — read this section when an agent needs
|
|
259
259
|
> the full rationale, examples, and decision tree for a given rule.
|
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: skillopt
|
|
3
|
+
description: Microsoft's SkillOpt trains skill files via rollouts + reflection against a frozen target model. Install with `pip install skillopt` and produce a `best_skill.md`. Always human-review.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# SkillOpt Workflow
|
|
7
|
+
|
|
8
|
+
## Overview
|
|
9
|
+
|
|
10
|
+
SkillOpt (microsoft/SkillOpt, MIT-licensed) is a Microsoft Research Python tool that **optimizes skill files for a specific target model**. Given a starting `SKILL.md`, a target model (frozen), a benchmark of inputs, and a reward signal, SkillOpt runs a rollout → reflect → aggregate → select → update → evaluate loop and emits a `best_skill.md` artifact that scores higher against the held-out validation gate.
|
|
11
|
+
|
|
12
|
+
It is *not* a general prompt editor and *not* a free-form rewriting tool. It is a closed-loop optimizer: it changes the skill text, runs the agent under the new skill against the benchmark, measures reward, and keeps mutations that improve the score.
|
|
13
|
+
|
|
14
|
+
**Core Principle:** SkillOpt trains a skill file the way you'd fine-tune a model — with a held-out validation gate, a frozen target, and a reward signal — not by hand-tweaking prose. Don't commit the output without human review; treat `best_skill.md` as a candidate, not a verdict.
|
|
15
|
+
|
|
16
|
+
## When to Use
|
|
17
|
+
|
|
18
|
+
- A skill's quality is measurably degrading on a benchmark (success rate dropping, regression in eval)
|
|
19
|
+
- You have a benchmark of representative inputs and a reward signal (task success / rubric score / pass-rate)
|
|
20
|
+
- You want to A/B a new skill variant against the current one against the same target model
|
|
21
|
+
- A skill has grown organically past ~200 lines and you suspect prose bloat is hurting recall
|
|
22
|
+
|
|
23
|
+
Decision flow:
|
|
24
|
+
|
|
25
|
+
```
|
|
26
|
+
Skill quality question?
|
|
27
|
+
→ Do you have a benchmark + reward + frozen target model? → no → DON'T RUN SKILLOPT (you can't measure without them)
|
|
28
|
+
→ Is the skill underperforming on the gate? → no → no work to do
|
|
29
|
+
→ Yes to both → see the 6-phase loop below
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
## When NOT to Use
|
|
33
|
+
|
|
34
|
+
- **No benchmark, no reward, no frozen target.** SkillOpt needs all three. If any is missing, the loop has nothing to optimize against — you'll burn compute on something you can't evaluate. Compose the benchmark first or skip SkillOpt entirely.
|
|
35
|
+
- **For a one-off skill polish.** SkillOpt is a multi-rollout loop with token cost on the order of dozens to hundreds of completions. For "rewrite this paragraph to be clearer," edit the skill by hand. Reach for SkillOpt when you have evidence the skill underperforms and a way to measure the fix.
|
|
36
|
+
- **On a non-frozen target model.** If the target model is updating daily, the rollout scores are non-reproducible and the held-out gate stops meaning anything. Pin the model version and don't retrain against a moving target.
|
|
37
|
+
- **Without human review of `best_skill.md`.** SkillOpt optimizes the reward signal — that is not the same as "is this skill correct, well-bounded, and aligned with what we want the agent to do." A skill that scores higher on the rubric can still be narrower, more brittle, or subtly off-vibe. Always review before committing.
|
|
38
|
+
|
|
39
|
+
## The 6-Phase Loop
|
|
40
|
+
|
|
41
|
+
```
|
|
42
|
+
┌─ Phase 1: Rollout ─ run the agent under the current skill against benchmark inputs
|
|
43
|
+
│
|
|
44
|
+
├─ Phase 2: Reflect ─ for each rollout, capture which skill instructions helped vs hurt
|
|
45
|
+
│
|
|
46
|
+
├─ Phase 3: Aggregate ─ across all rollouts, find recurring failure modes and structural patterns
|
|
47
|
+
│
|
|
48
|
+
├─ Phase 4: Select ─ choose one mutation hypothesis (a concrete edit) to try next
|
|
49
|
+
│
|
|
50
|
+
├─ Phase 5: Update ─ apply the mutation to produce a candidate skill
|
|
51
|
+
│
|
|
52
|
+
└─ Phase 6: Evaluate ─ run the candidate against the HELD-OUT validation set; keep if it improves, drop if not
|
|
53
|
+
↑ │
|
|
54
|
+
└────────────── next iteration reuses the kept variant ─────────┘
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
### Phase 1 — Rollout
|
|
58
|
+
|
|
59
|
+
Run the agent under the **frozen target model** with the current skill against the benchmark. Capture outputs + per-input scores. Don't peek at the validation set during this phase.
|
|
60
|
+
|
|
61
|
+
### Phase 2 — Reflect
|
|
62
|
+
|
|
63
|
+
For each rollout, ask: which lines of the skill contributed to the success or failure? Mark helpful / neutral / harmful spans. This is the closest you get to a teacher signal without a human labeler.
|
|
64
|
+
|
|
65
|
+
### Phase 3 — Aggregate
|
|
66
|
+
|
|
67
|
+
Across all rollouts this iteration, find the recurring failure modes. If 4 of 5 failures on a particular input pattern come from the same missing instruction in the skill, that's the highest-leverage next mutation.
|
|
68
|
+
|
|
69
|
+
### Phase 4 — Select
|
|
70
|
+
|
|
71
|
+
Pick **one** mutation to try. SkillOpt is not a search-and-replace shotgun; it's one edit per round. Co-edits entangle effects and make the held-out gate stop carrying signal.
|
|
72
|
+
|
|
73
|
+
### Phase 5 — Update
|
|
74
|
+
|
|
75
|
+
Apply the mutation. Save the candidate as a sibling file (e.g. `candidate-3.md`); never overwrite `SKILL.md` mid-loop.
|
|
76
|
+
|
|
77
|
+
### Phase 6 — Evaluate
|
|
78
|
+
|
|
79
|
+
Run the candidate against the **held-out validation set** (never been seen during Phases 1–5). If it improves the held-out gate, promote it to the next iteration's starting point. If it regresses, drop it and try a different mutation. Do not cherry-pick from a known input; the held-out gate is what makes the loop trustworthy.
|
|
80
|
+
|
|
81
|
+
## How Bizar Agents Use It
|
|
82
|
+
|
|
83
|
+
```bash
|
|
84
|
+
# 1. Install (do this once per machine)
|
|
85
|
+
pip install skillopt
|
|
86
|
+
|
|
87
|
+
# 2. Set up a benchmark directory (you write this — SkillOpt doesn't generate it)
|
|
88
|
+
# benchmark/
|
|
89
|
+
# ├── inputs.jsonl # one prompt per line, hold-out set included
|
|
90
|
+
# ├── reward.py # your scoring fn: takes (prompt, completion) → float
|
|
91
|
+
# └── target_model.json # the frozen target model spec (id + version pin)
|
|
92
|
+
|
|
93
|
+
# 3. Run the loop against a draft skill
|
|
94
|
+
python -m skillopt.cli optimize \
|
|
95
|
+
--skill config/skills/thinking-some-skill/SKILL.md \
|
|
96
|
+
--benchmark ./benchmark \
|
|
97
|
+
--out ./skillopt-out
|
|
98
|
+
|
|
99
|
+
# 4. Inspect the output before committing anything
|
|
100
|
+
ls skillopt-out/
|
|
101
|
+
cat skillopt-out/best_skill.md
|
|
102
|
+
diff config/skills/thinking-some-skill/SKILL.md skillopt-out/best_skill.md
|
|
103
|
+
|
|
104
|
+
# 5. If the diff is genuinely better (held-out improvement + human review passes),
|
|
105
|
+
# replace the source. Otherwise discard and tune the benchmark.
|
|
106
|
+
cp skillopt-out/best_skill.md config/skills/thinking-some-skill/SKILL.md
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
We do **not** vendor SkillOpt into the Bizar JS harness — the tool lives in Python (`pip install skillopt`), runs offline, and writes a markdown artifact. The Bizar loop is: write draft → run SkillOpt → human review `best_skill.md` → commit if better.
|
|
110
|
+
|
|
111
|
+
## Pitfalls
|
|
112
|
+
|
|
113
|
+
- **Don't retrain daily.** The target model is frozen for a reason; if your benchmark doesn't drift, the skill shouldn't either. Bumping SkillOpt runs on every PR turns the skill into a moving target the agent can't build a stable intuition against.
|
|
114
|
+
- **Don't bypass the held-out gate.** It's tempting to "peek" at validation items to bias the mutation — that converts the loop into a curve-fit on a small set and the held-out number stops meaning anything. Hold-out integrity is the whole point.
|
|
115
|
+
- **Never auto-commit `best_skill.md`.** It is a *candidate*. The reward signal optimizes task success; it does not optimize for tone, scope, or alignment with sibling skills in the same family. A/B test in shadow, then promote deliberately.
|
|
116
|
+
- **Don't use SkillOpt on skills you don't have a benchmark for.** If the input space and reward are unknown, you're optimizing prose against nothing and will learn something — just not something useful.
|
|
117
|
+
- **Don't optimize `thinking-*` skills together as one bundle.** They share front-matter shape but address different problem types; co-optimizing entangles their effects and makes individual improvements hard to attribute.
|
|
118
|
+
|
|
119
|
+
## Relationship to Bizar's Existing Skill Pipeline
|
|
120
|
+
|
|
121
|
+
| Concern | Owner |
|
|
122
|
+
|---|---|
|
|
123
|
+
| What skills ship | `config/skills/<name>/SKILL.md` (canonical, provisioned) |
|
|
124
|
+
| Provision to user | `cli/provision.mjs:syncConfigExtras` + `writeBizarSkillLock` |
|
|
125
|
+
| Skill discovery at runtime | `~/.claude/skills/<name>/SKILL.md` (provisioned mirror) |
|
|
126
|
+
| Offline quality loop | SkillOpt (this skill) — run on a draft, produce `best_skill.md`, human-promote |
|
|
127
|
+
| Editing prose by hand | You + your editor — fine for one-off tweaks, don't loop it |
|
|
128
|
+
|
|
129
|
+
SkillOpt is a **research-quality tool** for the rare case where you have the benchmark, the reward, and the time. For 90% of skill edits, just write it well.
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: thinking-archetypes
|
|
3
|
+
description: Use when the same problem keeps recurring despite fixes, growth stalled with no obvious cause, or a quick fix made things worse—match it to a known structural pattern instead of re-diagnosing.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Systems Archetypes
|
|
7
|
+
|
|
8
|
+
## Overview
|
|
9
|
+
|
|
10
|
+
Systems archetypes (Peter Senge, "The Fifth Discipline") are recurring structural patterns. Like design patterns in software, once you recognize one you can predict where it leads and where to intervene. Most stubborn, recurring problems aren't unique—they're an instance of a known pattern, and the pattern names the leverage point.
|
|
11
|
+
|
|
12
|
+
**Core Principle:** A recurring problem usually has a structural pattern. Match the pattern and the intervention follows.
|
|
13
|
+
|
|
14
|
+
## Trigger Card
|
|
15
|
+
|
|
16
|
+
When the same problem keeps recurring despite multiple fixes:
|
|
17
|
+
|
|
18
|
+
1. **Describe the problem neutrally** — symptoms, what's been tried, what keeps coming back.
|
|
19
|
+
2. **Match to an archetype** using the Quick Reference Card below (Fixes That Fail, Shifting the Burden, Limits to Growth, Tragedy of the Commons, Escalation, Success to the Successful, Growth and Underinvestment).
|
|
20
|
+
3. **Intervene at the structure, not the symptom** — use the Key Question for that archetype.
|
|
21
|
+
|
|
22
|
+
If no archetype fits after a genuine look, don't force one — drop to `thinking-systems` and map from scratch. For a first-time one-off problem, just fix it.
|
|
23
|
+
|
|
24
|
+
## When to Use
|
|
25
|
+
|
|
26
|
+
- The same problem keeps recurring despite multiple "fixes"
|
|
27
|
+
- A quick fix made things worse over time
|
|
28
|
+
- Growth stalled without an obvious cause
|
|
29
|
+
- A shared resource (CI, staging, a cache, a service quota) keeps degrading
|
|
30
|
+
- Two efforts are escalating against each other (duplicated services, alert/threshold arms races)
|
|
31
|
+
|
|
32
|
+
```
|
|
33
|
+
Problem keeps recurring despite fixes? → match an archetype
|
|
34
|
+
Quick fix made it worse? → match an archetype
|
|
35
|
+
Growth hit an invisible ceiling? → match an archetype
|
|
36
|
+
Shared resource degrading? → match an archetype
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
## When NOT to Use
|
|
40
|
+
|
|
41
|
+
- **You need to map the system from scratch** (components, interactions, emergent behavior) — use `thinking-systems` instead. Archetypes are pattern-matching shortcuts for recurring structures; if the system is unfamiliar, map it first, then see if an archetype fits.
|
|
42
|
+
- **You've found the pattern and need to know where to intervene** — use `thinking-leverage-points` (Meadows' hierarchy). Archetypes name the structure; leverage-points tell you where in that structure to act for maximum effect.
|
|
43
|
+
- A first-time, one-off problem with a clear cause → just fix it; there's no recurring structure to match.
|
|
44
|
+
- No candidate archetype fits after a genuine look → don't force one. Drop to `thinking-systems` and map the actual structure from scratch.
|
|
45
|
+
- You need to localize a specific faulty codepath → use a hypothesis differential, not pattern-matching.
|
|
46
|
+
|
|
47
|
+
## Quick Reference Card
|
|
48
|
+
|
|
49
|
+
Match the symptom to the pattern; the Key Question points at the leverage.
|
|
50
|
+
|
|
51
|
+
| Archetype | Structure | Recognize it by | Key Question (leverage) |
|
|
52
|
+
|-----------|-----------|-----------------|-------------------------|
|
|
53
|
+
| **Fixes That Fail** | Quick fix relieves symptom but a delayed side effect makes it worse | "We fixed this last quarter, why is it back?"; the fix needs ever-larger doses | "What side effect will this fix create?" |
|
|
54
|
+
| **Shifting the Burden** | A symptomatic workaround is used instead of the fundamental fix, creating dependency | Permanent workarounds; "we know the real fix but have no time"; the real capability is atrophying | "What capability are we not building by leaning on this workaround?" |
|
|
55
|
+
| **Limits to Growth** | A reinforcing growth loop hits a balancing constraint | Strong growth that plateaued; more effort, diminishing returns; a resource maxed out | "What will limit us at 10x scale?" |
|
|
56
|
+
| **Tragedy of the Commons** | Each actor gains by using a shared resource; collective overuse depletes it | Shared CI/staging/cache/quota degrading; everyone optimizes locally; no clear owner | "Who owns the long-term health of this resource?" |
|
|
57
|
+
| **Escalation** | Two parties react to each other's moves in a competitive spiral | "They did X so we must do Y"; arms-race dynamics; each side feels defensive | "Can we change the game instead of playing it harder?" |
|
|
58
|
+
| **Success to the Successful** | Initial success grants more resources, compounding advantage and starving alternatives | Past success is the main predictor of new investment; experiments starved to feed the incumbent | "Are we starving future successes to feed current ones?" |
|
|
59
|
+
| **Growth and Underinvestment** | Demand grows toward a limit; capacity investment is delayed until performance degrades into crisis | Reactive investment after the incident; chronic "good enough for now" | "What fails if we grow 50% without adding capacity now?" |
|
|
60
|
+
|
|
61
|
+
## Software Examples (per archetype)
|
|
62
|
+
|
|
63
|
+
- **Fixes That Fail:** Adding servers to mask a memory leak → cost spirals, leak remains. Silencing alerts → system degrades until outage.
|
|
64
|
+
- **Shifting the Burden:** Ops manually intervening forever instead of automating; feature flags hiding bugs that never get fixed.
|
|
65
|
+
- **Limits to Growth:** Velocity drops as the codebase grows (complexity limit); adding engineers doesn't speed delivery (coordination overhead).
|
|
66
|
+
- **Tragedy of the Commons:** Teams overload shared CI → everyone's builds slow; cloud costs spike as nobody cleans up resources.
|
|
67
|
+
- **Escalation:** Microservice teams duplicating functionality in response to each other; ever-stricter alert thresholds chasing each other.
|
|
68
|
+
- **Success to the Successful:** Legacy monolith gets all attention so the new architecture never matures.
|
|
69
|
+
- **Growth and Underinvestment:** Database not upgraded until it crashes under load; security investment only after a breach.
|
|
70
|
+
|
|
71
|
+
## Diagnosis Process
|
|
72
|
+
|
|
73
|
+
1. **Describe the problem neutrally** — symptoms, who/what is involved, what's been tried, what keeps recurring. No blame.
|
|
74
|
+
2. **Sketch the loops** — what causes what; where reinforcing (amplifying) loops are, where balancing (limiting) loops are, and where the delays sit.
|
|
75
|
+
3. **Match to an archetype** using the Quick Reference Card. If two fit, the system may be layered—note both.
|
|
76
|
+
4. **Intervene at the structure, not the symptom** — use the Key Question to find the fundamental solution, the constraint to remove, or the missing/ delayed feedback to add. If none match, stop and use `thinking-systems`.
|
|
77
|
+
|
|
78
|
+
## Verification Checklist
|
|
79
|
+
|
|
80
|
+
- [ ] Problem described without blame
|
|
81
|
+
- [ ] Loops sketched (reinforcing, balancing, delays)
|
|
82
|
+
- [ ] Matched to a specific archetype (or escalated to `thinking-systems` because none fit)
|
|
83
|
+
- [ ] Intervention targets structure, not the recurring symptom
|
|
84
|
+
- [ ] Considered the side effects of the intervention
|
|
85
|
+
|
|
86
|
+
## Senge's Wisdom
|
|
87
|
+
|
|
88
|
+
"Structures of which we are unaware hold us prisoner. Once we can see them, they no longer have the same hold on us."
|
|
89
|
+
|
|
90
|
+
The pattern continues until someone sees it and changes the structure driving it—not the behaviors, but what produces them.
|
|
@@ -0,0 +1,267 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: thinking-bayesian
|
|
3
|
+
description: Use when interpreting a test result, metric, or new evidence and you risk over-reacting to it. State the base rate first, then update belief by the likelihood ratio.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Bayesian Reasoning
|
|
7
|
+
|
|
8
|
+
## Overview
|
|
9
|
+
Bayesian thinking provides a framework for updating beliefs based on new evidence. Rather than treating beliefs as binary (true/false), it recognizes degrees of confidence that should shift as evidence accumulates. This approach, rooted in Bayes' Theorem, helps avoid both overconfidence and underreaction to new information.
|
|
10
|
+
|
|
11
|
+
**Core Principle:** State the base rate *before* you look at the evidence, then update. The single most common error is anchoring on a vivid result and skipping the prior — a positive test for a rare condition is usually still a false alarm.
|
|
12
|
+
|
|
13
|
+
> **Redirect:** For nearly all uncertainty-reasoning tasks, prefer `thinking-probabilistic` — it covers forecasting with ranges, calibration, and uncertainty communication without requiring explicit priors and likelihood ratios. Reserve this skill for the narrow case where you have a specific, quantified prior AND a specific piece of evidence to combine via Bayes' rule. If you only need a rough updated number, the base-rate-then-likelihood-ratio trigger below is enough — don't run the full theorem.
|
|
14
|
+
|
|
15
|
+
## Trigger Card
|
|
16
|
+
|
|
17
|
+
When interpreting a test result, metric, or new evidence where overreaction is a risk:
|
|
18
|
+
|
|
19
|
+
1. **State the base rate first** — what was the probability before the evidence?
|
|
20
|
+
2. **Estimate the likelihood ratio** — how much more expected is the evidence under the hypothesis vs. not?
|
|
21
|
+
3. **Update:** posterior odds = prior odds × likelihood ratio.
|
|
22
|
+
|
|
23
|
+
If the base rate is very low (rare condition), a positive result is usually still a false alarm. Always start with the prior.
|
|
24
|
+
|
|
25
|
+
## When to Use
|
|
26
|
+
- Estimating probabilities or likelihoods
|
|
27
|
+
- Interpreting test results or metrics
|
|
28
|
+
- Making decisions with incomplete information
|
|
29
|
+
- Evaluating competing hypotheses
|
|
30
|
+
- Learning from experiments or A/B tests
|
|
31
|
+
- Diagnosing problems with uncertain causes
|
|
32
|
+
- Predicting outcomes based on historical data
|
|
33
|
+
|
|
34
|
+
Decision flow:
|
|
35
|
+
```
|
|
36
|
+
Uncertain about something? → yes → Have prior belief? → yes → New evidence? → APPLY BAYESIAN UPDATE
|
|
37
|
+
↘ no → Establish base rate first
|
|
38
|
+
↘ no → Standard analysis may suffice
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
## When NOT to Use
|
|
42
|
+
|
|
43
|
+
- **You're reasoning under uncertainty and don't have a specific prior + evidence pair.** Use `thinking-probabilistic` instead — it handles forecasting with ranges, calibration, and uncertainty communication without requiring formal Bayesian machinery.
|
|
44
|
+
- **The evidence is conclusive or directly observable.** If you can read the logs, run the query, or look up the answer, do that — don't dress up a near-certainty as a probability.
|
|
45
|
+
- **No meaningful prior exists** and you'd just be inventing numbers. A fabricated base rate gives false rigor; say the prior is unknown instead.
|
|
46
|
+
- **The decision is the same at any plausible posterior.** If you'd act identically whether the probability is 40% or 70%, skip the update and act.
|
|
47
|
+
- **You need to express uncertainty as a calibrated range**, not a point update from a single piece of evidence — that's `thinking-probabilistic` territory.
|
|
48
|
+
|
|
49
|
+
## Key Concepts
|
|
50
|
+
|
|
51
|
+
### Prior Probability
|
|
52
|
+
Your belief BEFORE seeing new evidence:
|
|
53
|
+
```
|
|
54
|
+
P(H) = probability that hypothesis H is true
|
|
55
|
+
|
|
56
|
+
Example: Before any symptoms, what's the probability someone has disease X?
|
|
57
|
+
Use base rate: If 1 in 1000 people have it, P(disease) = 0.001
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
### Likelihood
|
|
61
|
+
How probable is the evidence IF the hypothesis is true?
|
|
62
|
+
```
|
|
63
|
+
P(E|H) = probability of seeing evidence E, given H is true
|
|
64
|
+
|
|
65
|
+
Example: If someone HAS the disease, what's the probability of a positive test?
|
|
66
|
+
If test is 99% sensitive: P(positive|disease) = 0.99
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
### Posterior Probability
|
|
70
|
+
Your belief AFTER seeing the evidence:
|
|
71
|
+
```
|
|
72
|
+
P(H|E) = updated probability of H, given you observed E
|
|
73
|
+
|
|
74
|
+
This is what Bayes' Theorem calculates.
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
## Bayes' Theorem
|
|
78
|
+
|
|
79
|
+
```
|
|
80
|
+
P(E|H) × P(H)
|
|
81
|
+
P(H|E) = ─────────────────────────
|
|
82
|
+
P(E)
|
|
83
|
+
|
|
84
|
+
Where:
|
|
85
|
+
P(H|E) = posterior (what we want)
|
|
86
|
+
P(E|H) = likelihood (how expected is evidence if H true)
|
|
87
|
+
P(H) = prior (initial belief)
|
|
88
|
+
P(E) = total probability of evidence
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
### Intuitive Form
|
|
92
|
+
|
|
93
|
+
```
|
|
94
|
+
Posterior odds = Prior odds × Likelihood ratio
|
|
95
|
+
|
|
96
|
+
If evidence is 10x more likely under H than under not-H,
|
|
97
|
+
your odds should shift by factor of 10.
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
## The Process
|
|
101
|
+
|
|
102
|
+
### Step 1: Establish Your Prior
|
|
103
|
+
What did you believe before this evidence?
|
|
104
|
+
- Use base rates when available
|
|
105
|
+
- Be explicit about uncertainty
|
|
106
|
+
- Don't anchor on 50% just because you're unsure
|
|
107
|
+
|
|
108
|
+
```
|
|
109
|
+
Question: Will this feature increase conversion?
|
|
110
|
+
Prior: Based on similar features, ~30% succeed significantly
|
|
111
|
+
P(success) = 0.30
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
### Step 2: Assess the Evidence
|
|
115
|
+
How strong is this evidence? Consider:
|
|
116
|
+
- How likely is this evidence if hypothesis is TRUE?
|
|
117
|
+
- How likely is this evidence if hypothesis is FALSE?
|
|
118
|
+
- What's the ratio?
|
|
119
|
+
|
|
120
|
+
```
|
|
121
|
+
Evidence: Early A/B test shows 5% lift (p=0.08)
|
|
122
|
+
P(this result | feature works) = 0.60 (moderately expected)
|
|
123
|
+
P(this result | feature doesn't work) = 0.15 (possible but less likely)
|
|
124
|
+
Likelihood ratio = 0.60 / 0.15 = 4x
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
### Step 3: Update Your Belief
|
|
128
|
+
Apply the likelihood ratio to your prior:
|
|
129
|
+
|
|
130
|
+
```
|
|
131
|
+
Prior odds: 0.30 / 0.70 = 0.43
|
|
132
|
+
Likelihood ratio: 4x
|
|
133
|
+
Posterior odds: 0.43 × 4 = 1.72
|
|
134
|
+
Posterior probability: 1.72 / (1 + 1.72) = 0.63
|
|
135
|
+
|
|
136
|
+
Updated belief: 63% confidence feature will succeed
|
|
137
|
+
(up from 30% prior)
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
### Step 4: Iterate as More Evidence Arrives
|
|
141
|
+
Yesterday's posterior becomes today's prior:
|
|
142
|
+
|
|
143
|
+
```
|
|
144
|
+
New evidence: Week 2 shows lift holding at 4.5%
|
|
145
|
+
Prior (from step 3): 0.63
|
|
146
|
+
[Repeat update process]
|
|
147
|
+
New posterior: 0.78
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
## Common Applications
|
|
151
|
+
|
|
152
|
+
### Interpreting Test Results
|
|
153
|
+
```
|
|
154
|
+
Scenario: Test for rare disease (1 in 10,000 prevalence)
|
|
155
|
+
Test: 99% sensitive, 99% specific
|
|
156
|
+
|
|
157
|
+
Prior: P(disease) = 0.0001
|
|
158
|
+
If positive test:
|
|
159
|
+
P(positive|disease) = 0.99
|
|
160
|
+
P(positive|no disease) = 0.01
|
|
161
|
+
P(positive) = 0.99 × 0.0001 + 0.01 × 0.9999 ≈ 0.0101
|
|
162
|
+
|
|
163
|
+
Posterior: P(disease|positive) = (0.99 × 0.0001) / 0.0101 ≈ 0.0098
|
|
164
|
+
|
|
165
|
+
Even with 99% accurate test, positive result only means ~1% chance of disease!
|
|
166
|
+
Base rate dominates when condition is rare.
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
### Debugging
|
|
170
|
+
```
|
|
171
|
+
Bug report: Users see error X
|
|
172
|
+
Prior beliefs:
|
|
173
|
+
P(database issue) = 0.20
|
|
174
|
+
P(network issue) = 0.30
|
|
175
|
+
P(code bug) = 0.40
|
|
176
|
+
P(user error) = 0.10
|
|
177
|
+
|
|
178
|
+
Evidence: Error happens only on mobile
|
|
179
|
+
P(mobile-only | database) = 0.05
|
|
180
|
+
P(mobile-only | network) = 0.30
|
|
181
|
+
P(mobile-only | code bug) = 0.60
|
|
182
|
+
P(mobile-only | user error) = 0.40
|
|
183
|
+
|
|
184
|
+
Update: Code bug becomes most likely (posterior ~0.55)
|
|
185
|
+
Next step: Investigate mobile-specific code paths
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
### Project Estimation
|
|
189
|
+
```
|
|
190
|
+
Prior: Based on similar projects, P(on-time) = 0.40
|
|
191
|
+
|
|
192
|
+
Evidence 1: Team is experienced with this stack
|
|
193
|
+
Likelihood ratio: 1.5x → Posterior: 0.50
|
|
194
|
+
|
|
195
|
+
Evidence 2: Requirements are unclear
|
|
196
|
+
Likelihood ratio: 0.6x → Posterior: 0.38
|
|
197
|
+
|
|
198
|
+
Evidence 3: Critical dependency has risk
|
|
199
|
+
Likelihood ratio: 0.7x → Posterior: 0.30
|
|
200
|
+
|
|
201
|
+
Final estimate: 30% chance of on-time delivery
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
## Mental Shortcuts
|
|
205
|
+
|
|
206
|
+
### Strong vs Weak Evidence
|
|
207
|
+
| Evidence Type | Typical Likelihood Ratio |
|
|
208
|
+
|---------------|-------------------------|
|
|
209
|
+
| Definitive proof | 100x+ |
|
|
210
|
+
| Strong evidence | 10-100x |
|
|
211
|
+
| Moderate evidence | 3-10x |
|
|
212
|
+
| Weak evidence | 1.5-3x |
|
|
213
|
+
| Noise | ~1x (no update) |
|
|
214
|
+
|
|
215
|
+
### When to Update Significantly
|
|
216
|
+
Update strongly when:
|
|
217
|
+
- Evidence is surprising under your current belief
|
|
218
|
+
- Evidence comes from reliable source
|
|
219
|
+
- Evidence is specific to your hypothesis
|
|
220
|
+
|
|
221
|
+
Update weakly when:
|
|
222
|
+
- Evidence is expected regardless of hypothesis
|
|
223
|
+
- Source has unknown reliability
|
|
224
|
+
- Evidence is circumstantial
|
|
225
|
+
|
|
226
|
+
### Base Rate Neglect (Avoid This)
|
|
227
|
+
Common error: Ignoring prior probability when evidence arrives
|
|
228
|
+
```
|
|
229
|
+
Wrong: "Positive test = probably have disease"
|
|
230
|
+
Right: "Positive test shifts probability, but base rate matters"
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
## Calibration Check
|
|
234
|
+
|
|
235
|
+
### Are You Well-Calibrated?
|
|
236
|
+
Track predictions and outcomes:
|
|
237
|
+
- Of things you said were "70% likely," did ~70% happen?
|
|
238
|
+
- If you're always overconfident, widen your uncertainty
|
|
239
|
+
- If you're always underconfident, trust your assessments more
|
|
240
|
+
|
|
241
|
+
### Confidence Levels
|
|
242
|
+
| Stated Confidence | Should Mean |
|
|
243
|
+
|-------------------|------------|
|
|
244
|
+
| 50% | Coin flip |
|
|
245
|
+
| 70% | Would bet 2:1 |
|
|
246
|
+
| 90% | Would bet 9:1 |
|
|
247
|
+
| 99% | Would bet 99:1 |
|
|
248
|
+
|
|
249
|
+
## Verification Checklist
|
|
250
|
+
- [ ] Established explicit prior probability (not just "I think...")
|
|
251
|
+
- [ ] Assessed likelihood ratio of evidence
|
|
252
|
+
- [ ] Applied update mathematically (not just "more/less likely")
|
|
253
|
+
- [ ] Considered base rates for rare events
|
|
254
|
+
- [ ] Checked for base rate neglect
|
|
255
|
+
- [ ] Documented reasoning for future calibration
|
|
256
|
+
|
|
257
|
+
## Key Questions
|
|
258
|
+
- "What was my belief before this evidence?"
|
|
259
|
+
- "How likely is this evidence if my belief is true? If false?"
|
|
260
|
+
- "What's the likelihood ratio?"
|
|
261
|
+
- "Am I anchoring on the evidence and ignoring base rates?"
|
|
262
|
+
- "How would I bet on this? At what odds?"
|
|
263
|
+
|
|
264
|
+
## Kahneman's Warning
|
|
265
|
+
"People tend to assess the relative importance of issues by the ease with which they are retrieved from memory—and this is largely determined by the extent of coverage in the media."
|
|
266
|
+
|
|
267
|
+
Don't let vivid evidence override base rates. A plane crash doesn't make flying more dangerous than driving, even though it feels that way.
|