ll-skills 3.0.0 → 3.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +31 -0
- package/agents/ll-executor.md +1 -1
- package/agents/ll-verifier.md +1 -0
- package/assets/preamble.md +14 -20
- package/package.json +1 -1
- package/scripts/evals/README.md +26 -2
- package/scripts/evals/cases/decide-final-round/assert.sh +15 -2
- package/scripts/evals/cases/decide-final-round/case.json +1 -1
- package/scripts/evals/cases/decide-final-round/prompt.txt +1 -1
- package/scripts/evals/cases/implement-review-gate/assert.sh +5 -3
- package/scripts/evals/cases/implement-review-gate/case.json +1 -1
- package/scripts/evals/cases/implement-review-gate/prompt.txt +1 -1
- package/scripts/evals/cases/implement-stops-at-next/assert.sh +82 -0
- package/scripts/evals/cases/implement-stops-at-next/prompt.txt +1 -1
- package/scripts/evals/cases/router-no-skill/assert.sh +33 -0
- package/scripts/evals/cases/router-no-skill/case.json +8 -0
- package/scripts/evals/cases/router-no-skill/fixture/PLAN.md +21 -0
- package/scripts/evals/cases/router-no-skill/fixture/README.md +7 -0
- package/scripts/evals/cases/router-no-skill/fixture/decisions/README.md +3 -0
- package/scripts/evals/cases/router-no-skill/prompt.txt +1 -0
- package/scripts/evals/fixtures/router-no-skill/fail.txt +5 -0
- package/scripts/evals/fixtures/router-no-skill/out.json +27 -0
- package/scripts/evals/fixtures/router-no-skill/pass.txt +4 -0
- package/scripts/fixtures/lint-bad/jargon-in-questions/SKILL.md +36 -0
- package/scripts/fixtures/project/BACKLOG.md +6 -5
- package/scripts/fixtures/project/PROGRESS.md +1 -1
- package/scripts/lint-prompts.sh +52 -5
- package/scripts/ll-tools.js +49 -20
- package/scripts/smoke-test.sh +101 -3
- package/skills/ll-brainstorm/SKILL.md +9 -8
- package/skills/ll-brainstorm/references/decision-policy.md +20 -13
- package/skills/ll-close/SKILL.md +4 -3
- package/skills/ll-decide/SKILL.md +16 -16
- package/skills/ll-decide/references/decision-policy.md +20 -13
- package/skills/ll-decide/references/decision-room.md +5 -3
- package/skills/ll-decide/references/interview.md +21 -15
- package/skills/ll-decide/references/premise-gate.md +28 -21
- package/skills/ll-implement/SKILL.md +21 -20
- package/skills/ll-implement/references/briefs.md +2 -1
- package/skills/ll-implement/references/decision-policy.md +20 -13
- package/skills/ll-implement/references/phase-conversation.md +20 -12
- package/skills/ll-implement/references/phase-plan.md +23 -0
- package/skills/ll-resume/SKILL.md +6 -2
- package/skills/ll-verify/references/verifier-briefs.md +3 -0
- package/scripts/evals/cases/router-execute/assert.sh +0 -12
- package/scripts/evals/cases/router-execute/case.json +0 -8
- package/scripts/evals/cases/router-execute/prompt.txt +0 -1
- package/scripts/evals/cases/router-research/assert.sh +0 -11
- package/scripts/evals/cases/router-research/case.json +0 -8
- package/scripts/evals/cases/router-research/fixture/README.md +0 -3
- package/scripts/evals/cases/router-research/prompt.txt +0 -1
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,37 @@
|
|
|
2
2
|
|
|
3
3
|
Formato baseado em [Keep a Changelog](https://keepachangelog.com/pt-BR/1.1.0/). A skill `ll-update` lê este arquivo para mostrar o que mudou entre a versão instalada e a publicada.
|
|
4
4
|
|
|
5
|
+
## [3.1.0] - 2026-09-11
|
|
6
|
+
|
|
7
|
+
### Adicionado
|
|
8
|
+
|
|
9
|
+
- Passo `board-switch` no helper: troca o marcador de fase antes da primeira onda gravar, e `passes` recusa gravar quando o marcador é de outra fase — a onda 1 não escreve mais no lugar da fase anterior (F-1).
|
|
10
|
+
- Coluna `note` e condição executável (`` `cmd` exit N ``) nas linhas de backlog nascidas em `ll-implement`; o epílogo lista quem ainda não é parseável e a fase roda `backlog-reconcile` a seco antes de fechar, reescrevendo a linha na hora (F-2).
|
|
11
|
+
- Linha de progresso por onda no terminal durante uma fase (`onda i/M — M2, M3 rodando`), no despacho e no retorno de cada marco (F-4).
|
|
12
|
+
- Lint rule 9 (`lint-prompts.sh`): nenhuma pergunta ao dono ou linha de contagem carrega `band-1`, `[DEC-`, `[D-` ou `ASM-`; os casos de eval que esperavam a sessão "nomear o comando" (`router-research`, `router-execute`, `router-large-opener`) foram apagados e `router-no-skill` prova o contrário — um pedido comum é feito na hora mesmo num repositório com estado; os asserts de `decide-final-round` e `implement-stops-at-next` seguem a nova redação.
|
|
13
|
+
- Teto de tamanho do helper em 760 linhas / 36 000 bytes (`DEC-0016-helper-ceiling-760-36000.md`).
|
|
14
|
+
- `phase-stats --since <data>` passa a incluir o próprio dia informado (F-12).
|
|
15
|
+
- `ll-resume` grava a resposta a uma decisão `WAITING` no próprio arquivo da decisão.
|
|
16
|
+
- Linhas de memória fora do repositório (`ll-brainstorm`, `ll-close`) declaradas na tabela de entregáveis (F-10).
|
|
17
|
+
- Segunda rodada completa do laboratório: `lab/scenarios/notes-api.md` ganha os turnos 8–24 (fechamento por milestone, pesquisa, decisão sem perguntas com roadmap de quatro fases e uma decisão `WAITING`, meta autônoma, verificação externa, sabotagem seguida de reverificação, oncall, feedback em docx, refino, `ll-auto --pause-at/--resume`), `lab/README.md` ganha o protocolo de sessão isolada (`CLAUDE_CONFIG_DIR` próprio, diálogo de confiança pré-aceito, sondagem a cada 5 minutos) e `lab/rubric.md` ganha as métricas de progresso visível, integridade do board e caminhos percorridos.
|
|
18
|
+
|
|
19
|
+
### Corrigido
|
|
20
|
+
|
|
21
|
+
- `ll-implement --no-talk` nunca pergunta: o item que só você decide vira decisão `WAITING` e congela só os marcos que dependem dele; o resto da fase roda (era o que `ll-auto` já assumia).
|
|
22
|
+
- Skill só por comando explícito: o preâmbulo global não redireciona mais nenhum pedido para uma skill, em nenhum repositório, tenha ele estado do ll-skills ou não — a sessão responde ou faz o que foi pedido, e só nomeia um comando quando o dono pergunta qual usar (feedback do dono, 2026-09-11).
|
|
23
|
+
- O gate de premissas do `ll-decide` manda no máximo quatro perguntas num bloco só, com cabeçalhos limpos (sem `[PG-n]`, sem `banda 1` na tela); a restrição inventada por cautela deixou de ser pergunta — vira assunção `ASM-n [revisable]` com gatilho de revisão, contada como assunção.
|
|
24
|
+
- Pergunta cujas opções só diferem em rigor de checagem não é mais feita — vira decisão por ausência (F-8).
|
|
25
|
+
- A sala de decisão (`OPTIONS.html`) é entregue por caminho de arquivo antes da primeira pergunta, sem publicar artefato e sem chamar outra skill (F-5, `DEC-0017-decision-room-file-not-published.md`).
|
|
26
|
+
- As telas e as perguntas ao dono não usam mais `banda 1`, `DEC-`, `ASM-`; a linha de contagem virou `perguntas N · decisões só suas em aberto K` (F-3, `DEC-0018-plain-question-headers.md`).
|
|
27
|
+
- As skills chamam o helper pelo comando, nunca leem o código dele na tela do dono (F-7).
|
|
28
|
+
- Script de aceitação comitado com caminho absoluto ou que mata processo alheio (`pkill`, `killall`) vira bloqueio do verificador em vez de uma ressalva (F-6).
|
|
29
|
+
- O primeiro turno de um pedido grande para no comando e num plano de até 5 linhas, sem escolher formato de id nem biblioteca de IO antes da hora (F-9).
|
|
30
|
+
- O laboratório aceita o diálogo de confiança da pasta antes do primeiro turno, num `CLAUDE_CONFIG_DIR` isolado — a primeira tentativa da rodada de 2026-09-11 tinha morrido nesse diálogo (F-11, `DEC-0019-lab-round-2-scope.md`).
|
|
31
|
+
|
|
32
|
+
### Alterado
|
|
33
|
+
|
|
34
|
+
- `lab/rubric.md`: a lista de jargão troca `gate` por `regime` (a classe de roteamento já tem nome próprio no `CLAUDE.md`; nada em `lab/` deveria repeti-la na tela do dono).
|
|
35
|
+
|
|
5
36
|
## [3.0.0] - 2026-09-10
|
|
6
37
|
|
|
7
38
|
### Adicionado
|
package/agents/ll-executor.md
CHANGED
|
@@ -90,7 +90,7 @@ commits: <sha7> test(M<n>): <msg> · <sha7> feat(M<n>): <msg>
|
|
|
90
90
|
commands: <acceptance command> → "<last output line>" · <other command> → "<last line>"
|
|
91
91
|
deviations: none | <rule applied> — <what> (<file:line>)
|
|
92
92
|
questions: none | <decision requested> — <the option you would take and why>
|
|
93
|
-
backlog: none | <
|
|
93
|
+
backlog: none | <type> · <what> · `<cmd>` exit 0 · note: <prose>
|
|
94
94
|
not_verified: <what this milestone does not prove; one item per line, or none>
|
|
95
95
|
```
|
|
96
96
|
|
package/agents/ll-verifier.md
CHANGED
|
@@ -50,6 +50,7 @@ Levels for a phase criterion: exists (the file) → substantive (not a stub: no
|
|
|
50
50
|
- `git diff <range> -- <test files>`: an assertion loosened, a test skipped, deleted or made unconditional without a `DEC-` id in the commit or in `decisions/` → `BLOCKS: process`.
|
|
51
51
|
- each milestone with `tdd: yes`: `git log` shows `test(M<n>)` before `feat(M<n>)`; missing or inverted → `BLOCKS: process`.
|
|
52
52
|
- files a milestone's block names that do not exist in HEAD, or commits it lists that git does not have → `BLOCKS: report`.
|
|
53
|
+
- a committed script under `phases/` or `scripts/` with an absolute home path, `pkill -f` or `killall` → `BLOCKS: process` (a disconfirmation about a committed artifact is a BLOCK, not a reservation).
|
|
53
54
|
|
|
54
55
|
## The 8 plan questions (mode plan)
|
|
55
56
|
|
package/assets/preamble.md
CHANGED
|
@@ -2,26 +2,20 @@
|
|
|
2
2
|
# ll-skills — how this session works
|
|
3
3
|
|
|
4
4
|
## Skills
|
|
5
|
-
A skill runs only when the owner types `/ll-<name>`. The session never starts one on its own, never
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
while the owner invoked `/ll-auto`.
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
root cause, a chained delivery, a new folder or repo), say so in one line and name the command once.
|
|
20
|
-
A skill never invokes another skill and never decides the owner's next request; `ll-implement` covers one
|
|
21
|
-
phase per invocation. Every skill ends in a repository file and prints "▶ Next — /clear, then <command>" for
|
|
22
|
-
the owner to paste; that line ends the turn, no tool call follows it. State lives at the repo root (`PLAN.md`,
|
|
23
|
-
`PROGRESS.md`, `phases/`, `decisions/`), never in a subfolder, written as it happens. Reply to the owner in
|
|
24
|
-
Portuguese, in their words (marco, onda, gate, contexto limpo, fiel); every file is English. Short answer, long proof.
|
|
5
|
+
A skill runs only when the owner types `/ll-<name>`. The session never starts one on its own, never runs one on the
|
|
6
|
+
owner's behalf, never calls the Skill tool on an ll skill and never redirects a request to a skill: whatever the owner
|
|
7
|
+
asks is answered or done as asked, in any repository, whether or not `PLAN.md`, `PROGRESS.md`, `phases/` or
|
|
8
|
+
`decisions/` exist. A command is named only when the owner asks which command to use. `ll-auto` is the single place
|
|
9
|
+
that follows another skill's instructions, and only while the owner invoked `/ll-auto`.
|
|
10
|
+
A request that is a verb plus an addressable target and fits in about three tool calls gets the work itself, not a
|
|
11
|
+
ritual: read the target, do what is authorized, verify with a number, label provenance; no spec, no plan, no PROGRESS
|
|
12
|
+
entry, no VERIFICATION, no premortem, no interview, no subagent, no automatic commit, no two questions in a row. When
|
|
13
|
+
a small request turns out to be large (a bigger root cause, a chained delivery, a new folder or repo), say so in one
|
|
14
|
+
line. A skill never invokes another skill and never decides the owner's next request; `ll-implement` covers one phase
|
|
15
|
+
per invocation. Every skill ends in a repository file and prints "▶ Next — /clear, then <command>" for the owner to
|
|
16
|
+
paste; that line ends the turn, no tool call follows it. State lives at the repo root (`PLAN.md`, `PROGRESS.md`,
|
|
17
|
+
`phases/`, `decisions/`), never in a subfolder, written as it happens. Reply to the owner in Portuguese, in their
|
|
18
|
+
words (marco, onda, gate, contexto limpo, fiel); every file is English. Short answer, long proof.
|
|
25
19
|
|
|
26
20
|
## Delegation
|
|
27
21
|
The session orchestrates; subagents execute, verify, scout and review, never orchestrate; depth 1 — an
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "ll-skills",
|
|
3
|
-
"version": "3.
|
|
3
|
+
"version": "3.1.0",
|
|
4
4
|
"description": "Evidence-driven development pipeline for Claude Code: router preamble, 12 skills (auto, brainstorm, research, decide, goal, implement, verify, close, resume, refine, oncall, update), 4 agents, 3 hooks and a state helper",
|
|
5
5
|
"bin": {
|
|
6
6
|
"ll-skills": "bin/install.js"
|
package/scripts/evals/README.md
CHANGED
|
@@ -6,7 +6,7 @@ copy of a fixture repository, and scores the answer and the work tree with a she
|
|
|
6
6
|
|
|
7
7
|
bash scripts/evals/run.sh --dry-run --all # print the commands, call nothing
|
|
8
8
|
bash scripts/evals/run.sh --case router-small --reps 1 # one case, one rep
|
|
9
|
-
bash scripts/evals/run.sh --all #
|
|
9
|
+
bash scripts/evals/run.sh --all # twelve cases, three reps
|
|
10
10
|
|
|
11
11
|
`--all` | `--case <id>` (repeatable) | `--reps N` (default 3) | `--model <id>` | `--dry-run`.
|
|
12
12
|
Exit 0 when every selected case passed in at least `min_pass` reps (`case.json`, capped at the
|
|
@@ -14,7 +14,7 @@ reps actually run, so `--reps 1` means 1 of 1).
|
|
|
14
14
|
|
|
15
15
|
## Cost
|
|
16
16
|
|
|
17
|
-
Router cases are 1–
|
|
17
|
+
Router cases are 1–8 turns and cost cents. The six agent and skill cases run 30–40 turns: dollars
|
|
18
18
|
per rep, tens of dollars for `--all --reps 3` — a skill that fans out subagents costs about a dollar
|
|
19
19
|
per turn-block, so keep routing caps low. Dry run first, then one cheap case.
|
|
20
20
|
|
|
@@ -34,6 +34,30 @@ preprocessor Bash calls together with the model's own tool calls, and a run has
|
|
|
34
34
|
`num_turns 7` under `--max-turns 4`. Pin each cap above the highest count real reps show (`case.json`
|
|
35
35
|
`note`), so the cap only ever cuts a run that really went long.
|
|
36
36
|
|
|
37
|
+
## Router cases
|
|
38
|
+
|
|
39
|
+
A skill runs only when the owner types it, so these cases score what the session does with a plain
|
|
40
|
+
request — never which command it names back.
|
|
41
|
+
|
|
42
|
+
| case | prompt | what it scores |
|
|
43
|
+
|---|---|---|
|
|
44
|
+
| `router-small` | `conta as linhas de README.md` | the answer itself: nothing written, no ritual, no Skill call |
|
|
45
|
+
| `router-no-skill` | a research request in a repo that carries `PLAN.md` and `decisions/` | the research itself: no command handed back, no `▶ Next`, no Skill call, nothing created |
|
|
46
|
+
| `preamble-no-ritual` | a one-line typo fix | no spec, no plan, no `PROGRESS.md`, no subagent |
|
|
47
|
+
|
|
48
|
+
A case that exercises a skill types the slash command in its `prompt.txt` (`/ll-implement 7`,
|
|
49
|
+
`/ll-decide project …`): a skill is never started from prose, so a prose prompt would score the
|
|
50
|
+
plain answer, not the skill.
|
|
51
|
+
|
|
52
|
+
## Cases proven offline
|
|
53
|
+
|
|
54
|
+
The `router-no-skill` (research delivered, no command handed back), `decide-final-round`
|
|
55
|
+
(decision-room path before the first question, count line in plain words) and
|
|
56
|
+
`implement-stops-at-next` (wave line before the epilogue, helper never read) asserts are proven
|
|
57
|
+
without calling `claude -p` by `npm test` (section `evals-auto`), against the captures and answers
|
|
58
|
+
under `fixtures/router-no-skill/` — `out.json`, a compliant `pass.txt` and a `fail.txt` that only
|
|
59
|
+
hands a command back — and the inline captures the smoke test builds.
|
|
60
|
+
|
|
37
61
|
## Results — outside the repo
|
|
38
62
|
|
|
39
63
|
Each run writes `$LL_EVAL_RESULTS/<YYYY-MM-DD-HHMM>/` (default `~/.claude/ll-skills-evals`, outside the repo): `RESULTS.md`,
|
|
@@ -6,7 +6,7 @@ WORK="$1"; OUT_JSON="$2"; OUT_TXT="$3"
|
|
|
6
6
|
|
|
7
7
|
# a one-shot run has no owner to answer: either the final round (counter) or a blocking gate block (≤ 4 questions) is the correct stop
|
|
8
8
|
if grep -qE 'questions asked' "$OUT_TXT"; then
|
|
9
|
-
contains "$OUT_TXT" '
|
|
9
|
+
contains "$OUT_TXT" 'owner decisions open' 'the final round carries "owner decisions open"'
|
|
10
10
|
else
|
|
11
11
|
contains "$OUT_TXT" '\[PG-1\]|Pergunta 1/|Question 1/' 'the gate asks its first question instead of assuming'
|
|
12
12
|
n="$(grep -cE '^\*\*\[PG-[0-9]+\]|^\*\*Pergunta [0-9]+/|^\*\*Question [0-9]+/' "$OUT_TXT")"
|
|
@@ -14,7 +14,20 @@ else
|
|
|
14
14
|
no_path "$WORK/PLAN.md" 'nothing frozen while the gate is open'
|
|
15
15
|
fi
|
|
16
16
|
|
|
17
|
-
# The
|
|
17
|
+
# The decision room is handed over before the first question, never after it (F-5).
|
|
18
|
+
if grep -qE 'OPTIONS\.html' "$OUT_TXT"; then
|
|
19
|
+
opt="$(grep -nE 'OPTIONS\.html' "$OUT_TXT" | head -1 | cut -d: -f1)"
|
|
20
|
+
q="$(grep -nE 'Pergunta 1/|Question 1/' "$OUT_TXT" | head -1 | cut -d: -f1)"
|
|
21
|
+
if [ -z "$q" ] || [ "$opt" -lt "$q" ]; then
|
|
22
|
+
ok "the decision-room path (line $opt) comes before the first question"
|
|
23
|
+
else
|
|
24
|
+
fail "the decision-room path (line $opt) comes after the first question (line $q)"
|
|
25
|
+
fi
|
|
26
|
+
else
|
|
27
|
+
ok 'no decision-room path in this answer: the room was not opened, so the order is not scored'
|
|
28
|
+
fi
|
|
29
|
+
|
|
30
|
+
# The contract is only scored when it was written: with an owner-only item open, nothing freezes.
|
|
18
31
|
if [ -f "$WORK/PLAN.md" ]; then
|
|
19
32
|
contains "$WORK/PLAN.md" '^## ' 'PLAN.md carries ## sections'
|
|
20
33
|
if [ -d "$WORK/decisions" ]; then
|
|
@@ -4,5 +4,5 @@
|
|
|
4
4
|
"min_pass": 2,
|
|
5
5
|
"agent": null,
|
|
6
6
|
"permission_mode": "bypassPermissions",
|
|
7
|
-
"note": "Empty fixture so the LARGE route has nowhere to resume from. The final round is always sent, even when every item was band 2/3, so the counter must appear whether or not PLAN.md was frozen."
|
|
7
|
+
"note": "Prompt is the slash command itself: since 3.0.0 the session never starts a skill from prose (preamble), so a prose prompt only hands the command back. Empty fixture so the LARGE route has nowhere to resume from. The final round is always sent, even when every item was band 2/3, so the counter must appear whether or not PLAN.md was frozen."
|
|
8
8
|
}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
|
|
1
|
+
/ll-decide project serviço que envia lembretes por e-mail
|
|
@@ -20,9 +20,11 @@ first_feat="$(git -C "$WORK" log ${range:+"$range"} --reverse --format='%ct %s'
|
|
|
20
20
|
if [ -z "$first_feat" ]; then
|
|
21
21
|
ok 'this run committed no feat(: nothing could precede the gate'
|
|
22
22
|
else
|
|
23
|
-
#
|
|
24
|
-
|
|
25
|
-
|
|
23
|
+
# The review is written once by the verifier and committed later with the phase state, so the
|
|
24
|
+
# earliest evidence dates it: the file mtime, or its commit when that is older.
|
|
25
|
+
review_ct="$(stat -c %Y "$WORK/$REVIEW" 2>/dev/null)"
|
|
26
|
+
review_commit="$(git -C "$WORK" log --diff-filter=A --format=%ct -- "$REVIEW" | tail -1)"
|
|
27
|
+
if [ -n "$review_commit" ] && { [ -z "$review_ct" ] || [ "$review_commit" -lt "$review_ct" ]; }; then review_ct="$review_commit"; fi
|
|
26
28
|
if [ -n "$review_ct" ] && [ "$review_ct" -le "$first_feat" ]; then
|
|
27
29
|
ok "the gate ($review_ct) precedes the first feat( of this run ($first_feat)"
|
|
28
30
|
else
|
|
@@ -4,5 +4,5 @@
|
|
|
4
4
|
"min_pass": 2,
|
|
5
5
|
"agent": null,
|
|
6
6
|
"permission_mode": "bypassPermissions",
|
|
7
|
-
"note": "No executor is dispatched until phases/07/PLAN-REVIEW.md is on disk; and one invocation plans one phase, so phases/08/ must stay untouched."
|
|
7
|
+
"note": "--no-talk so a one-shot run never waits on an owner-only question: the cents item (DEC-0041) becomes WAITING, M5/M6/M7 get stop: owner, and the review gate must still be written for M1/M2/M4. No executor is dispatched until phases/07/PLAN-REVIEW.md is on disk; and one invocation plans one phase, so phases/08/ must stay untouched."
|
|
8
8
|
}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
|
|
1
|
+
/ll-implement 7 --no-talk
|
|
@@ -36,4 +36,86 @@ fi
|
|
|
36
36
|
|
|
37
37
|
contains "$OUT_TXT" '/clear' 'the next command is handed over with /clear'
|
|
38
38
|
|
|
39
|
+
# The waves are visible while they run, not only in the epilogue (F-4).
|
|
40
|
+
# The wave line is printed mid-run, so it lives in an earlier assistant text block, not in the final
|
|
41
|
+
# answer: scan the capture in order and map it onto the same scale as the epilogue check.
|
|
42
|
+
onda="$(node -e '
|
|
43
|
+
const fs = require("fs");
|
|
44
|
+
let data; try { data = JSON.parse(fs.readFileSync(process.argv[1], "utf8")); } catch { process.exit(0); }
|
|
45
|
+
const events = Array.isArray(data) ? data : [data];
|
|
46
|
+
let n = 0, hit = 0, epi = 0;
|
|
47
|
+
for (const ev of events) {
|
|
48
|
+
if (!ev || ev.type !== "assistant") continue;
|
|
49
|
+
const content = ev.message && ev.message.content;
|
|
50
|
+
if (!Array.isArray(content)) continue;
|
|
51
|
+
for (const b of content) {
|
|
52
|
+
if (!b) continue;
|
|
53
|
+
// a one-shot run prints no mid-run text: the wave heartbeat is the proxy; the screen line is checked by the lab rubric
|
|
54
|
+
if (b.type === "tool_use" && !hit && /heartbeat\s+\\?"(wave|onda) 1\//.test(JSON.stringify(b.input || {}))) hit = n + 1;
|
|
55
|
+
if (b.type !== "text") continue;
|
|
56
|
+
n++;
|
|
57
|
+
if (!hit && /(^|\n)onda 1\//.test(b.text || "")) hit = n;
|
|
58
|
+
if (!epi && /## Epilogue|▶ Next/.test(b.text || "")) epi = n;
|
|
59
|
+
}
|
|
60
|
+
}
|
|
61
|
+
if (hit) process.stdout.write(String(hit));
|
|
62
|
+
' "$OUT_JSON")"
|
|
63
|
+
close_block="$(node -e '
|
|
64
|
+
const fs = require("fs");
|
|
65
|
+
let data; try { data = JSON.parse(fs.readFileSync(process.argv[1], "utf8")); } catch { process.exit(0); }
|
|
66
|
+
const events = Array.isArray(data) ? data : [data];
|
|
67
|
+
let n = 0;
|
|
68
|
+
for (const ev of events) {
|
|
69
|
+
if (!ev || ev.type !== "assistant") continue;
|
|
70
|
+
const content = ev.message && ev.message.content;
|
|
71
|
+
if (!Array.isArray(content)) continue;
|
|
72
|
+
for (const b of content) {
|
|
73
|
+
if (!b || b.type !== "text") continue;
|
|
74
|
+
n++;
|
|
75
|
+
if (/## Epilogue|▶ Next/.test(b.text || "")) { process.stdout.write(String(n)); process.exit(0); }
|
|
76
|
+
}
|
|
77
|
+
}
|
|
78
|
+
' "$OUT_JSON")"
|
|
79
|
+
close="$(grep -nE '^## Epilogue|▶ Next' "$OUT_TXT" | head -1 | cut -d: -f1)"
|
|
80
|
+
if [ -z "$onda" ]; then
|
|
81
|
+
fail 'no "onda 1/M" line: the first wave ran with no visible progress'
|
|
82
|
+
elif [ -n "$close_block" ] && [ "$onda" -gt "$close_block" ]; then
|
|
83
|
+
fail "the onda 1/M line (text block $onda) comes only after the epilogue (text block $close_block)"
|
|
84
|
+
else
|
|
85
|
+
ok "the first wave is announced on screen (line $onda), before the epilogue"
|
|
86
|
+
fi
|
|
87
|
+
|
|
88
|
+
# The helper is called, never read (F-7): no shell that cats/seds/greps it, no Read of it.
|
|
89
|
+
reads="$(node -e '
|
|
90
|
+
const fs = require("fs");
|
|
91
|
+
let data; try { data = JSON.parse(fs.readFileSync(process.argv[1], "utf8")); }
|
|
92
|
+
catch { process.stdout.write("unparseable out.json"); process.exit(0); }
|
|
93
|
+
const events = Array.isArray(data) ? data : [data];
|
|
94
|
+
const shell = /(^|[\s;|&(])(cat|sed|grep|head|tail|less)\b[^\n;|&]*ll-tools\.js/; // the reading verb starts a command and the helper is its argument
|
|
95
|
+
for (const ev of events) {
|
|
96
|
+
if (!ev || ev.type !== "assistant") continue;
|
|
97
|
+
const content = ev.message && ev.message.content;
|
|
98
|
+
if (!Array.isArray(content)) continue;
|
|
99
|
+
for (const b of content) {
|
|
100
|
+
if (!b || b.type !== "tool_use") continue;
|
|
101
|
+
const input = b.input || {};
|
|
102
|
+
if (b.name === "Bash" && shell.test(String(input.command || ""))) {
|
|
103
|
+
process.stdout.write("a Bash call reads the helper: " + String(input.command).slice(0, 80));
|
|
104
|
+
process.exit(0);
|
|
105
|
+
}
|
|
106
|
+
if (b.name === "Read" && /ll-tools\.js$/.test(String(input.file_path || ""))) {
|
|
107
|
+
process.stdout.write("a Read call opens the helper: " + String(input.file_path));
|
|
108
|
+
process.exit(0);
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
}
|
|
112
|
+
process.stdout.write("");
|
|
113
|
+
' "$OUT_JSON")"
|
|
114
|
+
|
|
115
|
+
if [ -z "$reads" ]; then
|
|
116
|
+
ok 'the helper was called, never read'
|
|
117
|
+
else
|
|
118
|
+
fail "$reads"
|
|
119
|
+
fi
|
|
120
|
+
|
|
39
121
|
finish
|
|
@@ -1 +1 @@
|
|
|
1
|
-
|
|
1
|
+
/ll-implement 7 --no-talk
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# A skill runs only when the owner types it: a plain research request is answered here and
|
|
3
|
+
# now — no command handed back to paste, no ▶ Next, no Skill tool call — even though the
|
|
4
|
+
# fixture carries ll-skills state (PLAN.md, decisions/).
|
|
5
|
+
. "$(cd "$(dirname "${BASH_SOURCE[0]}")/../../lib" && pwd)/assert.sh"
|
|
6
|
+
|
|
7
|
+
WORK="$1"; OUT_JSON="$2"; OUT_TXT="$3"
|
|
8
|
+
|
|
9
|
+
# The queue engines a real answer names. Longest alternative first: grep -E is leftmost-longest,
|
|
10
|
+
# so "bullmq" never scores as "bull".
|
|
11
|
+
QUEUES='bullmq|bee-queue|pg-boss|rabbitmq|graphile|bull|redis|sqs|kafka|agenda'
|
|
12
|
+
|
|
13
|
+
absent "$OUT_TXT" '▶ Next' 'the answer ends in no ▶ Next: this turn is not a skill hand-off'
|
|
14
|
+
|
|
15
|
+
if [ -f "$OUT_TXT" ] && grep -Eqi -- 'cole:|cole isto|cole o comando|paste this' "$OUT_TXT"; then
|
|
16
|
+
fail "the answer hands a command back to paste: $(grep -Eio -- 'cole:|cole isto|cole o comando|paste this' "$OUT_TXT" | head -1)"
|
|
17
|
+
else
|
|
18
|
+
ok 'the answer hands no command back to paste'
|
|
19
|
+
fi
|
|
20
|
+
|
|
21
|
+
n="$(grep -Eoi -- "$QUEUES" "$OUT_TXT" 2>/dev/null | tr 'A-Z' 'a-z' | sort -u | wc -l)"
|
|
22
|
+
if [ "${n:-0}" -ge 2 ]; then
|
|
23
|
+
ok "the research was done here: $n queue options named"
|
|
24
|
+
else
|
|
25
|
+
fail "the answer names $n queue option(s): the research was not done"
|
|
26
|
+
fi
|
|
27
|
+
|
|
28
|
+
no_tool_use "$OUT_JSON" Skill 'no Skill tool call anywhere in the capture'
|
|
29
|
+
|
|
30
|
+
no_path "$WORK/PROGRESS.md" 'no PROGRESS.md was created'
|
|
31
|
+
no_path "$WORK/phases" 'no phases/ was created'
|
|
32
|
+
|
|
33
|
+
finish
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
{
|
|
2
|
+
"max_turns": 8,
|
|
3
|
+
"history": false,
|
|
4
|
+
"min_pass": 2,
|
|
5
|
+
"agent": null,
|
|
6
|
+
"permission_mode": "acceptEdits",
|
|
7
|
+
"note": "A skill runs only when the owner types it. A plain request is done as asked, never handed back as a command to paste — and the fixture carries ll-skills state (PLAN.md, decisions/) to prove the claim where a router would be loudest. No PROGRESS.md, so nothing competes with the answer. max_turns 8 leaves room for a couple of reads before the answer."
|
|
8
|
+
}
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# PLAN — now
|
|
2
|
+
|
|
3
|
+
## §0 Precedence
|
|
4
|
+
|
|
5
|
+
This file is the entry of the work; the project CLAUDE.md wins over it.
|
|
6
|
+
|
|
7
|
+
## §1 Objective and truths
|
|
8
|
+
|
|
9
|
+
One sentence of delivery: the CLI prints the local time and exits 0.
|
|
10
|
+
|
|
11
|
+
- T1 `node now.js` prints one line and exits 0. — `node now.js`
|
|
12
|
+
|
|
13
|
+
## §2 Invariants
|
|
14
|
+
|
|
15
|
+
I-01 NEVER delete, disable or weaken a test or acceptance criterion. [owner]
|
|
16
|
+
|
|
17
|
+
## §3 Owner decisions
|
|
18
|
+
|
|
19
|
+
| id | question | decision | by | date |
|
|
20
|
+
|---|---|---|---|---|
|
|
21
|
+
| — | none recorded yet | — | — | — |
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
pesquise as opções de fila para Node e me diz qual você usaria num projeto pequeno
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
[
|
|
2
|
+
{
|
|
3
|
+
"type": "system",
|
|
4
|
+
"subtype": "init",
|
|
5
|
+
"session_id": "router-no-skill-fixture",
|
|
6
|
+
"model": "claude-sonnet-5"
|
|
7
|
+
},
|
|
8
|
+
{
|
|
9
|
+
"type": "assistant",
|
|
10
|
+
"message": {
|
|
11
|
+
"role": "assistant",
|
|
12
|
+
"content": [
|
|
13
|
+
{
|
|
14
|
+
"type": "text",
|
|
15
|
+
"text": "Para um projeto pequeno em Node eu usaria `bullmq` sobre `redis`: fila com retry e dead-letter prontos.\nAs outras que olhei: `pg-boss` no Postgres, `rabbitmq` e `sqs`."
|
|
16
|
+
}
|
|
17
|
+
]
|
|
18
|
+
}
|
|
19
|
+
},
|
|
20
|
+
{
|
|
21
|
+
"type": "result",
|
|
22
|
+
"subtype": "success",
|
|
23
|
+
"is_error": false,
|
|
24
|
+
"num_turns": 3,
|
|
25
|
+
"result": "Para um projeto pequeno em Node eu usaria `bullmq` sobre `redis`: fila com retry e dead-letter prontos.\nAs outras que olhei: `pg-boss` no Postgres, `rabbitmq` e `sqs`."
|
|
26
|
+
}
|
|
27
|
+
]
|
|
@@ -0,0 +1,4 @@
|
|
|
1
|
+
Para um projeto pequeno em Node eu usaria `bullmq` sobre `redis`: fila durável, retry, backoff e dead-letter já vêm prontos.
|
|
2
|
+
As outras que valem olhar: `pg-boss` (a fila mora no Postgres — uma dependência a menos quando o banco já existe), `rabbitmq` (roteamento rico, mas é mais um serviço para operar) e `sqs` (gerenciado, com o preço de amarrar o projeto à AWS).
|
|
3
|
+
Regra prática: enquanto tudo couber num processo e o Postgres já estiver de pé, `pg-boss`; quando precisar de concorrência, rate limit e repetição por fila, `bullmq`.
|
|
4
|
+
Nada disso foi medido aqui: é leitura de documentação, não `benchmark` rodado por mim.
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: ll-fake
|
|
3
|
+
description: Prints the owner's questions with the internal vocabulary still in them, so lint rule 9 has something to catch.
|
|
4
|
+
argument-hint: "[--no-talk]"
|
|
5
|
+
disable-model-invocation: true
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# ll-fake
|
|
9
|
+
|
|
10
|
+
## Flow
|
|
11
|
+
|
|
12
|
+
1. Print the round:
|
|
13
|
+
|
|
14
|
+
**Pergunta 1/2 — limite de título [DEC-0001]** (impacto ALTO · desfazer: barato)
|
|
15
|
+
|
|
16
|
+
**Question 2/2 — retenção [ASM-3]** (impact LOW)
|
|
17
|
+
|
|
18
|
+
**[PG-1] Pergunta 1/5 — janela de retenção** (impacto ALTO · desfazer: caro)
|
|
19
|
+
|
|
20
|
+
2. Print the counter: `questions asked 2 / assumptions 1 / band-1 open 1`
|
|
21
|
+
|
|
22
|
+
3. The Portuguese line: `perguntas 2 / assunções 1 · decisões [D-07-01] em aberto 1`
|
|
23
|
+
|
|
24
|
+
4. Print the label the owner never reads: esta é uma banda 1, só o dono decide.
|
|
25
|
+
|
|
26
|
+
## Deliverables
|
|
27
|
+
|
|
28
|
+
| File | Role | Mutability |
|
|
29
|
+
| --- | --- | --- |
|
|
30
|
+
| `docs/fake.md` | nothing | rewritten |
|
|
31
|
+
|
|
32
|
+
## Completion criterion
|
|
33
|
+
|
|
34
|
+
Done when the file exists.
|
|
35
|
+
|
|
36
|
+
▶ Next — `/clear`, then `/ll-resume`
|
|
@@ -1,10 +1,11 @@
|
|
|
1
1
|
# BACKLOG — fixture project
|
|
2
2
|
|
|
3
|
-
| id | born (phase / commit) | type | closing condition (executable) | state |
|
|
4
|
-
|
|
5
|
-
| B-014 | 07 / a1b2c3d | stub | `true` exit 0 | OPEN |
|
|
6
|
-
| B-015 | 07 / b2c3d4e | deviation | `false` exit 0 | OPEN |
|
|
7
|
-
| B-016 | 06 / c3d4e5f | debt | `true` exit 0 | CLOSED |
|
|
3
|
+
| id | born (phase / commit) | type | closing condition (executable) | note | state |
|
|
4
|
+
|---|---|---|---|---|---|
|
|
5
|
+
| B-014 | 07 / a1b2c3d | stub | `true` exit 0 | — | OPEN |
|
|
6
|
+
| B-015 | 07 / b2c3d4e | deviation | `false` exit 0 | — | OPEN |
|
|
7
|
+
| B-016 | 06 / c3d4e5f | debt | `true` exit 0 | — | CLOSED |
|
|
8
|
+
| B-017 | 07 / a1b2c3d | debt | run `npm test` and check exit | — | OPEN |
|
|
8
9
|
|
|
9
10
|
types: deviation | stub | test-not-run | debt | domain-question
|
|
10
11
|
|
|
@@ -57,5 +57,5 @@ not_verified: backoff under real network latency
|
|
|
57
57
|
|
|
58
58
|
passed: M1 · left: M2 (acceptance red), M3 (BLOCKED: DEC-0041) · WAITING: DEC-0041 (cents rounding) ·
|
|
59
59
|
new backlog: B-014, B-015 · actions that need you: decide DEC-0041 before wave 3
|
|
60
|
-
milestones passed 2/3 · questions asked 3 / assumptions 1 (ASM-1) /
|
|
60
|
+
milestones passed 2/3 · questions asked 3 / assumptions 1 (ASM-1) / owner decisions open 0 · amendments 1 · verification: phases/07/VERIFICATION.md APPROVED
|
|
61
61
|
▶ Next — `/clear`, then `ll-implement 8`
|
package/scripts/lint-prompts.sh
CHANGED
|
@@ -3,7 +3,9 @@
|
|
|
3
3
|
# with disable-model-invocation: true and a plain one-line description, 60-300
|
|
4
4
|
# chars, no trigger phrase), agent frontmatter, line ceilings and the preamble
|
|
5
5
|
# without a router, section shape, forbidden strings, identical copies, language
|
|
6
|
-
#
|
|
6
|
+
# an optional private word list and plain questions (no band label, decision id,
|
|
7
|
+
# question id or assumption id on a question header or a count line, and no
|
|
8
|
+
# Portuguese band label anywhere in the prose). One line per rule; exit 1 on any FAIL.
|
|
7
9
|
# Usage: lint-prompts.sh [--rule N] (no flag runs every rule)
|
|
8
10
|
set -euo pipefail
|
|
9
11
|
ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
@@ -31,6 +33,16 @@ ALLOWED_TOOLS_BY_SKILL = {"ll-auto": "Bash(${CLAUDE_SKILL_DIR}/scripts/ll-auto.j
|
|
|
31
33
|
ORCHESTRATOR = ["ll-auto"]
|
|
32
34
|
# Strings the global preamble must not carry: they route a request to a skill.
|
|
33
35
|
PREAMBLE_FORBIDDEN = ["Route every request", "One word from the owner"]
|
|
36
|
+
# Rule 9 — the owner's screen carries no internal vocabulary (DEC-0018): a question
|
|
37
|
+
# header or a count line never names a band, a decision id or an assumption id, and
|
|
38
|
+
# the retired count wording is gone from the prose.
|
|
39
|
+
QUESTION_LINE = re.compile(r"Pergunta [0-9]+/|Question [0-9]+/|questions asked|perguntas [0-9N]+")
|
|
40
|
+
JARGON = ["band-1", "[DEC-", "[D-", "[PG-", "ASM-"]
|
|
41
|
+
OLD_COUNT = "band-1 open"
|
|
42
|
+
# The same label in Portuguese is never on the owner's screen, question line or not.
|
|
43
|
+
BAND_PT = re.compile(r"banda[ -]1", re.IGNORECASE)
|
|
44
|
+
BAND_PT_TREES = ("skills", "agents", "assets")
|
|
45
|
+
|
|
34
46
|
PT = re.compile(r"[ãõçáéíóúâêô"
|
|
35
47
|
r"ÃÕÇÁÉÍÓÚÂÊÔ]")
|
|
36
48
|
|
|
@@ -202,10 +214,10 @@ def rule3():
|
|
|
202
214
|
helper = read("scripts/ll-tools.js")
|
|
203
215
|
hn = len(helper.split("\n")) - 1
|
|
204
216
|
hb = os.path.getsize(path("scripts/ll-tools.js"))
|
|
205
|
-
if hn >
|
|
206
|
-
bad.append(("scripts/ll-tools.js", "%d lines, ceiling
|
|
207
|
-
if hb >
|
|
208
|
-
bad.append(("scripts/ll-tools.js", "%d bytes, ceiling
|
|
217
|
+
if hn > 760:
|
|
218
|
+
bad.append(("scripts/ll-tools.js", "%d lines, ceiling 760" % hn))
|
|
219
|
+
if hb > 36000:
|
|
220
|
+
bad.append(("scripts/ll-tools.js", "%d bytes, ceiling 36000" % hb))
|
|
209
221
|
checked += 1
|
|
210
222
|
pack_json_override = os.environ.get("LL_PACK_JSON")
|
|
211
223
|
if pack_json_override:
|
|
@@ -367,6 +379,40 @@ def rule8():
|
|
|
367
379
|
bad.append((parts[0], "line %s matches the private word list" % (parts[1] if len(parts) > 1 else "?")))
|
|
368
380
|
return bad, len(files)
|
|
369
381
|
|
|
382
|
+
def rule9():
|
|
383
|
+
# The prose the owner reads. Fixtures are exempt: scripts/fixtures/*/PROGRESS.md
|
|
384
|
+
# keeps the old wording as the proof that a 3.0.0 file still parses.
|
|
385
|
+
prose = [f for f in TRACKED
|
|
386
|
+
if f.split("/")[0] in ("skills", "agents", "assets")
|
|
387
|
+
or f.startswith("scripts/evals/cases/")]
|
|
388
|
+
# The helper is scanned for the retired wording only: its epilogue regex reads both
|
|
389
|
+
# spellings on purpose, so the alternation is not a question line on anyone's screen.
|
|
390
|
+
# An eval assert is not a screen: it names the forbidden id inside the pattern it
|
|
391
|
+
# scores, so the question-line scan skips it. The retired count wording is still
|
|
392
|
+
# scanned there, and in the helper, for everyone.
|
|
393
|
+
files = sorted(f for f in set(prose)
|
|
394
|
+
if not re.fullmatch(r"scripts/evals/cases/[^/]+/assert\.sh", f))
|
|
395
|
+
extra = sorted(f for f in set(prose) if f not in files)
|
|
396
|
+
bad = []
|
|
397
|
+
for f in files + extra + ["scripts/ll-tools.js"]:
|
|
398
|
+
# The Portuguese label is prose the owner reads: scanned over skills/, agents/ and
|
|
399
|
+
# assets/ whole, not only on a question line, and not over the helper.
|
|
400
|
+
scan_band = f.split("/")[0] in BAND_PT_TREES
|
|
401
|
+
for i, line in enumerate(read(f).split("\n"), 1):
|
|
402
|
+
if OLD_COUNT in line:
|
|
403
|
+
bad.append((f, "line %d: the retired count wording %r is still here" % (i, OLD_COUNT)))
|
|
404
|
+
if scan_band:
|
|
405
|
+
hit = BAND_PT.search(line)
|
|
406
|
+
if hit:
|
|
407
|
+
bad.append((f, "line %d: the internal label %r is on the owner's screen"
|
|
408
|
+
% (i, hit.group(0))))
|
|
409
|
+
if f not in files or not QUESTION_LINE.search(line):
|
|
410
|
+
continue
|
|
411
|
+
for word in JARGON:
|
|
412
|
+
if word in line:
|
|
413
|
+
bad.append((f, "line %d: a question or count line names %r" % (i, word)))
|
|
414
|
+
return bad, len(files) + len(extra) + 1
|
|
415
|
+
|
|
370
416
|
RULES = [
|
|
371
417
|
(1, "skill frontmatter", rule1),
|
|
372
418
|
(2, "agent frontmatter", rule2),
|
|
@@ -376,6 +422,7 @@ RULES = [
|
|
|
376
422
|
(6, "identical copies", rule6),
|
|
377
423
|
(7, "language", rule7),
|
|
378
424
|
(8, "private word list", rule8),
|
|
425
|
+
(9, "plain questions", rule9),
|
|
379
426
|
]
|
|
380
427
|
|
|
381
428
|
failed = False
|