@zenera/cli 1.1.24 → 1.1.26

package/templates/editor/.github/skills/zen-code-python/SKILL.md

@@ -0,0 +1,123 @@
+---
+name: zen-code-python
+description: Code an agent writes must be a reusable tool that can be remembered and re-run, never a one-off. Holds the coding rules (`references/code_python.md`) and decides where they go in the target project - a section of a dedicated code agent's system prompt, or a coding skill for generic agents that sometimes run code - how to adapt them, and what a prompt or skill for a code-writing agent must never say. Load before writing, editing or reviewing any agent prompt, skill or house rule for an agent that writes or runs code - any agent holding `sandbox:*` or `run_command` - and before shipping a script inside a skill.
+---
+
+# Code an agent writes is a tool
+
+A script an agent writes is worth more than the answer it printed. Written
+generically - every changing value an argument, a name that says what it does -
+it can be committed to memory as a `file` node and found again by the next
+similar question, which runs it instead of writing it again. Written as a
+one-off - this customer's id, this month, this path, as literals - it can only
+ever answer the question that produced it, and memory fills up with near-copies
+that never fold together.
+
+Nothing in the runtime enforces this, and `zen check` does not report it. The
+rules reach an agent only through what you write into the target project.
+
+## The reference
+
+`references/code_python.md` is the rule set, in wording that has been tested on
+models. It is a source to build from, not a file to copy. Start from its
+sentences - copying and then editing them works; rewriting them from scratch in
+your own words loses the mechanical phrasing that makes them obeyed.
+
+**Keep in substance, and never soften:**
+
+- every value that would change for the next similar question is a named
+  argument, never a literal
+- the file is named for what it does, and describes itself through `--help`
+- the result goes to stdout, a produced file to an `--output` path
+- the script is written to a file and the file is run - no `python -c`, no
+  heredoc
+- before using the output, the script is checked against a different question
+  of the same kind
+- with memory: search before writing, commit the working script after
+
+**Adapt to the project:**
+
+- the interpreter and language, if the sandbox image runs something other than
+  Python
+- where scripts are written - `/workspace/scripts/` unless the project already
+  has a place
+- libraries the image carries and the agent should prefer, and the output
+  format the project expects (JSON, CSV, a table)
+- an example invocation from this project's own domain, values passed as
+  arguments
+- the memory steps: keep them when the agent holds `memory_search` /
+  `memory_commit`, drop them when it never will. In a skill several agents may
+  load, keep them gated on the tool, as the reference does
+
+## Decide where the rules go
+
+List the agents that hold `sandbox:*` or `run_command`, and read each one's
+prompt for what its job is. Then:
+
+| The project has                                                                          | Write                                                                                                          |
+| ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
+| A **dedicated code agent** - a runner, an executor, an analyst whose answers are scripts | A `## Writing code` section in that agent's `agents/prompts/<name>.md`                                         |
+| **No dedicated code agent** - generic agents that sometimes run code                     | A coding skill, `agents/skills/code_python/SKILL.md`                                                           |
+| Both                                                                                     | The prompt section, and the generic agents hand the code work to that agent rather than writing it - see below |
+| No agent that writes or runs code                                                        | Nothing                                                                                                        |
+
+**A dedicated code agent gets the rules in its system prompt.** It writes code
+on nearly every turn, so the rules are part of its job description: a skill
+would cost a load on every case, and preloading one is the same text in the
+prefix with an extra file to keep in step. Put the section after the agent's
+identity and procedure, before its prohibitions.
+
+**Generic agents get a skill.** Most of their turns write no code, so the rules
+should cost nothing until one does. The `description` is the routing key - name
+the condition, not the topic:
+
+```yaml
+description: Before writing, changing or running any Python script - how to write it as a reusable tool rather than a one-off.
+```
+
+Leave it in the agent's index; do not preload it. If `allow` is used, add
+`code_python` to it for every agent holding `sandbox:*`.
+
+**With both, one copy.** Generic agents that can reach a code agent should
+`fork:` to it for an answer, or hand off to it, rather than writing code
+themselves - then the rules have one home and the scripts one author. Grant
+them no `sandbox:*` for code work. If they must keep a shell of their own, they
+need the skill too, and its rules are the same sentences as the prompt section:
+a second wording is how one copy ends up softer than the other, and the softer
+one is the one obeyed.
+
+Run `zen check` after any change to `agents.yaml` or a skill.
+
+## Writing anything else around it
+
+- **One home per agent.** Once an agent has the rules - in its prompt or in its
+  skill - no house rule, other skill or prompt line restates them. A prompt that
+  needs to point at the skill names it: _"Follow `code_python` for every
+  script."_
+- **Never add an exemption.** _"Hard-coding is fine for a quick one-off check"_
+  is a permission: every probe is a one-off by construction, so the exception
+  admits exactly what the rule forbids. The same goes for `python -c`.
+- **Never show a one-off as an example.** A prompt or skill that shows
+  `python -c "..."`, or a script with a literal id, date or path in its body,
+  teaches the shape the rule forbids. Every example invocation passes its values
+  as named arguments:
+  `python /workspace/scripts/list_overdue_invoices.py --customer-id 42`.
+- **Scripts shipped inside a skill** (`agents/skills/<name>/scripts/`, §3.4.1 of
+  the copilot instructions) follow the same rules: named arguments, a `--help`,
+  the result on stdout, an `--output` path under `/workspace` for any file,
+  since `/skills` is read-only.
+- **Memory is what makes the reuse happen.** A code-writing agent without
+  memory still writes reusable scripts, but cannot find them in a later session.
+  Say so if the project expects reuse and that agent has no `memory:`.
+
+## Review
+
+- [ ] Every agent that writes or runs code has the rules - a prompt section for
+      a dedicated code agent, the `code_python` skill reachable for the others
+- [ ] Every rule under _Keep in substance_ is there, none softened, and the
+      memory steps match whether the agent holds the memory tools
+- [ ] Generic agents beside a dedicated code agent hand code work to it, or
+      carry the same wording - not a second paraphrase
+- [ ] No other prompt, skill or house rule restates the rules, adds an
+      exemption, or shows `python -c` or a script with literal values
+- [ ] Every script shipped in a skill takes its values as named arguments

package/templates/editor/.github/skills/zen-code-python/references/code_python.md

@@ -0,0 +1,48 @@
+# Writing Python scripts
+
+Every script you write is a tool for the next question like this one, not an
+answer to this one. Write it once, generically, so it can be remembered and run
+again unchanged.
+
+## Before writing
+
+If you hold `memory_search`, search for a script that already answers this kind
+of question, and load the best match with `memory_load`. If it fits, run it with
+this question's arguments and write nothing new. If it needs a change, change
+it the way the rest of this skill says - usually by adding an argument - and
+commit the new version as superseding the old one.
+
+## Writing
+
+1. **Solve the kind of question, not this one.** Anything that would differ the
+   next time - inputs, ids, names, paths, urls, dates and date ranges, filters,
+   thresholds, limits, the output path - is a named `argparse` argument such as
+   `--resource-id`, never a literal in the code. A default is allowed only for a
+   value that is the same for every question.
+2. **Name the file for what it does**, verb first: `list_overdue_invoices.py`,
+   `compare_quarterly_revenue.py`. Never `script.py`, `test.py`, `tmp.py` or a
+   name that only makes sense in this conversation.
+3. **Describe it.** Line one is a docstring saying what question it answers. The
+   same sentence is the `argparse` description, and every argument has a
+   `help=`, so `--help` is all a later reader needs.
+4. **Print the result to stdout** and anything else to stderr. Exit non-zero
+   with a message saying what failed when it cannot answer. A file it produces
+   goes to the path given by `--output`.
+5. **Take credentials from environment variables by name.** Never write a key,
+   token or password into the code or an argument default.
+6. **Use only what the sandbox image already has.** Do not `pip install` from
+   the script or before running it.
+7. **Write it to a file and run the file**:
+   `python /workspace/scripts/<name>.py --resource-id 42`. Never run Python
+   with `python -c` or a heredoc.
+
+## Before using its output
+
+- Run it, and fix the script - not the output - when it fails or prints
+  something wrong.
+- Read your own code against a different question of the same kind. If
+  answering it would mean editing the script rather than passing other
+  arguments, a value is still hard-coded: make it an argument and run again.
+- If you hold `memory_commit`, commit the working script as a `file` node. Its
+  text says what kind of question it answers and names its arguments, so that a
+  later search for a similar question finds it.

package/templates/editor/.github/skills/zen-dataset/SKILL.md

@@ -0,0 +1,253 @@
+---
+name: zen-dataset
+description: Keep a project's evaluation dataset in `dataset/` with `zen meta dataset` - every case read from its sources with a class, a complexity, tags, a verbatim rubric and an anchor to the section it came from. Covers extracting a training set of any shape into a proposal, applying it as a revision, finding which cases their sources moved under (drift) and re-reading only those sections, settling gone and uncovered sections, overriding a field by hand, leaving notes and verdicts on a case, reading a case's history and the sessions that touched it, and drawing samples by criteria (class, complexity, tag, rubric, revision, verdict, note kind, source, text) stratified, weighted and seeded. Load before building or refreshing a dataset, before running /dataset, before choosing which cases to run, and before writing a note or verdict on a case.
+---
+
+# The dataset
+
+A dataset is the project's set of real queries, each with what it should be
+judged by. It lives in `dataset/` at the project root, it is committed, and it
+outlives any one tuning. **Only `zen meta dataset` writes to it.** A hand edit
+skips the history, the revision, the session stamp and the lock, and the next
+drift check will not see it.
+
+Reading a source and deciding what is a case is your work. Everything after
+that - what changed, which revision, which cases restart, what to run next - is
+the command's.
+
+## What is stored
+
+```
+dataset/
+    manifest.json      revision, a hash per source file, the ignored sections
+    revisions.jsonl    one line per revision: added, updated, retired, restarted
+    cases/<id>.json    each case as it is now
+    log/<id>.jsonl     each case's history: every change, every note, who made it
+```
+
+A case:
+
+```json
+{
+    "id": "planning-organize-day",
+    "rev": 3,
+    "status": "active",
+    "class": "planning",
+    "complexity": "complex",
+    "tags": ["calendar", "mail"],
+    "input": "organize my day",
+    "rubric": [
+        { "id": "r1", "text": "calls api /mail/list" },
+        { "id": "r2", "text": "proposes a schedule, does not just list both" }
+    ],
+    "expected": "A time-ordered plan naming the two conflicting meetings.",
+    "source": {
+        "file": "SPECIFICATION.md",
+        "anchor": { "heading": ["Examples", "Query: organize my day"] }
+    }
+}
+```
+
+- **`rev`** is the dataset revision that last changed the case. A run that
+  records `{id, rev}` can always be read back against the rubric it ran under:
+  `zen meta dataset show <id>@<rev>`.
+- **Rubric ids** stay with their line. An unchanged line keeps its `r<n>`; a
+  new line gets a new one; a note that says `r2 failed` keeps meaning the same
+  line.
+- **Nothing is deleted.** A removed case is `retired`: out of every sample,
+  still readable, its id never reused.
+
+## Extracting cases
+
+Read the source yourself and write the structure out; do not write a parser.
+Large sources are read in parts, and the count is checked against the source at
+the end.
+
+| In the source                                                    | In the case                                                          |
+| ---------------------------------------------------------------- | -------------------------------------------------------------------- |
+| A heading like `## Query:` and a quoted string                   | One case, `input` is the string                                      |
+| A bullet list under `### Rubric:`                                | `rubric` - **verbatim**, one entry per bullet                        |
+| An embedded image, a `![](...)` link, an audio or PDF attachment | A media part in `input`, path relative to the proposal file          |
+| A paragraph of context before the query                          | `notes`, not `input` - unless the agent is meant to see it           |
+| The expected answer                                              | `expected`, one string; it is context for grading, not a diff target |
+| Nothing but a query                                              | A case with no `rubric`                                              |
+
+- **The rubric is the source's words.** A paraphrased rubric grades the
+  paraphrase. A wrong rubric is fixed in the source and re-read; when the
+  source is not yours to edit, `update --rubric-edit ... --override` and say why.
+- **Class** is a distinction the source already makes - a section, an agent,
+  an API, a task type. Do not invent a taxonomy.
+- **Complexity** rates the trajectory the query demands, not the sentence:
+
+| Level     | What it means                                                               |
+| --------- | --------------------------------------------------------------------------- |
+| `simple`  | One tool call or none; one agent; the answer is a lookup or a restatement   |
+| `medium`  | Several calls that depend on each other, or one delegation, or one artefact |
+| `complex` | Fan-out, multi-agent, ambiguity to resolve, or a plan that can go wrong     |
+
+- **Tags** are free words for what cuts across classes - `mail`, `pagination`,
+  `impossible`. Use the source's own words; a tag on one case is noise.
+- **Ids** are letters, digits, dot, dash and underscore, `class-short-slug`,
+  and **stable**: a case read again from the same section keeps its id. A new
+  id for an old case restarts it and orphans its history.
+- **`input` is exactly what `zen run batch` accepts**: a string, or a list of
+  parts - a string, `{ "text": "…" }`, or `{ "image" | "audio" | "video" | "file": "<path or url>" }`.
+
+### Anchors
+
+Every case read from a file names the section it came from, so the command can
+find that section again with no model involved:
+
+| Source                 | Anchor                                   | The section                                       |
+| ---------------------- | ---------------------------------------- | ------------------------------------------------- |
+| Markdown               | `{ "heading": "Examples > Query: one" }` | from the heading to the next one at its level     |
+| JSON or YAML           | `{ "pointer": "/cases/3" }`              | that value; key order and formatting do not count |
+| Anything else, or none | no `anchor`                              | the whole file                                    |
+
+A heading path needs only its last name while that is unique in the file; add
+a parent to tell two apart. Headings inside code fences are not headings. An
+anchor that does not resolve to exactly one section is refused at apply.
+
+### The proposal
+
+Write the cases to a scratch file - `.tmp/dataset/proposal.json` - as
+`{ "cases": [ … ] }`. Media paths in it resolve against that file. Then:
+
+```sh
+zen meta dataset apply .tmp/dataset/proposal.json --dry-run
+zen meta dataset apply .tmp/dataset/proposal.json --why "first extraction of SPECIFICATION.md"
+```
+
+A whole proposal **retires every active case it does not name**. A refresh of
+a few sections is `--partial`: only the cases it names are touched. Read the
+dry run before writing - an unexpected `retired` or an id you did not mean to
+change is caught there and nowhere else.
+
+The old `finetune/dataset.json` (`cases` or `samples`) applies as it stands.
+
+## When the sources move
+
+```sh
+zen meta dataset drift
+```
+
+A source file whose hash is unchanged is not opened. In a file that moved, each
+case's own section is compared, so a typo three sections away restarts nothing.
+
+| State       | Means                                                 | Do                                                                    |
+| ----------- | ----------------------------------------------------- | --------------------------------------------------------------------- |
+| `changed`   | the case's section text differs                       | re-read that section; put the case in a `--partial` proposal          |
+| `gone`      | its anchor no longer finds exactly one section        | `retire` it, or `reanchor` it if the section was only renamed         |
+| `uncovered` | a section at the cases' level that no case comes from | new cases in the proposal, or `ignore <file> --heading "…" --why "…"` |
+| `unsettled` | a file never recorded, because a section is uncovered | settle its uncovered sections                                         |
+
+Re-reading a `changed` section and proposing the same case is fine: if what it
+asks, expects and is judged by did not change, nothing is revised - only its
+stored hash moves.
+
+**What restarts a case.** A change to `input`, `rubric` or `expected` - the
+question or how it is judged - lists the case under `restarted` in that
+revision; results from its earlier revision are about a different question. A
+change to `class`, `complexity`, `tags`, `notes` or `source` revises the case
+without restarting it.
+
+## By hand
+
+```sh
+zen meta dataset update <id> --set class=search --set complexity=medium --why "…"
+zen meta dataset update <id> --rubric-edit r2="proposes a schedule" --override --why "…"
+zen meta dataset update <id> --rubric-add "says when it is impossible" --rubric-drop r3 --why "…"
+zen meta dataset update <id> --tag-add impossible --why "…"
+zen meta dataset retire <id>... --why "…"        # restore <id>... to undo
+zen meta dataset reanchor <id> --heading "Examples > Query: plan" --why "section renamed"
+```
+
+`--set <field>=` with nothing after the `=` clears that field. `--override` keeps the
+change through every later re-extraction; without it the next apply of that
+section writes the source's version back.
+
+## Notes, verdicts, history
+
+```sh
+zen meta dataset note <id>... -m "…" [--kind graded|analyze|difficult|rubric-suspect|observation] \
+    [--run <run>] [--verdict right|wrong|void] [--rubric r1=pass,r2=fail]
+zen meta dataset log [id...]
+```
+
+A note is about the case's **current** revision and is recorded with it; after
+the case is revised, the note stays in its history but stops counting. A note
+with a verdict defaults to `graded`. Leave a `rubric-suspect` note when a case
+fails for a reason the rubric cannot be right about - the dataset prompt lists
+them until someone fixes the source.
+
+Every change and note records who made it: the `zen meta` session, the prompt
+it ran (`/dataset`, `/analyze`, …) and the host. `log` prints
+`resume: zen meta resume <project> <session>` for every session this machine
+still holds.
+
+## Drawing a sample
+
+```sh
+zen meta dataset sample [filters] [-n <count>] [--seed <n>] [--by <fields>] \
+    [--weight <stratum>=<w>] [--order <keys>] [--format ids|batch|cases|json] [-o <file>]
+```
+
+**Filters** - all must hold; a repeated flag means any of its values:
+
+| Flag                                     | Keeps                                                       |
+| ---------------------------------------- | ----------------------------------------------------------- |
+| `--class`, `--complexity`, `--tag`       | those values (`unclassified`, `unrated` for none)           |
+| `--status active\|retired\|all`          | default `active`                                            |
+| `--rubric yes\|no`, `--expected yes\|no` | cases with or without one                                   |
+| `--id <glob>`, `--exclude <glob>`        | ids matching, ids not matching                              |
+| `--ids-from <file>`                      | ids listed - plain, or json with `ids`, `batch`, `cases`    |
+| `--changed-since <rev>`                  | revised after that revision                                 |
+| `--restarted`                            | restarted by the latest revision (or the `--at` one)        |
+| `--at <rev>`                             | the dataset as it stood then                                |
+| `--verdict right\|wrong\|void`           | the latest verdict on the current revision                  |
+| `--never-graded`                         | no verdict on the current revision                          |
+| `--note <kind>`, `--graded-in <run>`     | has a note of that kind, or one from that run               |
+| `--source <glob>`, `--anchor <glob>`     | read from that file, or that section                        |
+| `--grep <text>`                          | input or rubric contains it (`--regex`, `--case-sensitive`) |
+
+A value the dataset does not have - a misspelt class - is an error that lists
+the real ones, never an empty sample.
+
+**The draw.** Cases are grouped into strata by `--by` (default `class`; also
+`complexity`, `tag`, `source`, `rubric`, `verdict`, several joined by commas,
+or `none`). Inside a stratum they are ordered by `--order` (default
+`rubric,complexity`: graded first, then complex, medium, simple; also
+`verdict` - wrong first, `rev` - newest first, `id`), ties broken by `--seed`.
+Strata take turns, `--weight alpha=2` giving one twice the turns. The whole
+order is laid out before `-n` cuts it, so **a larger `-n` keeps every case a
+smaller one chose**, and the same filters, seed and revision draw the same cases
+on any machine.
+
+**Output.** `ids` (default) one per line; `batch` - `{ "batch": [{ id, input }] }`
+for `zen run batch`, **never a rubric**; `cases` - whole cases with rubrics, for
+a grader; `json` - the ids with the criteria, seed, revision and per-stratum
+counts, which is the record that reproduces the sample. Media paths are written
+relative to the `-o` file. The per-stratum table goes to stderr.
+
+```sh
+# a cross-section of 12 graded cases
+zen meta dataset sample --rubric yes -n 12 --format batch -o runs/r1/cases.json
+# everything the latest revision restarted
+zen meta dataset sample --restarted --by none --format batch -o runs/r2/cases.json
+# what is still wrong, worst classes first
+zen meta dataset sample --verdict wrong --order complexity --weight search=2
+# the whole dataset, rubrics included, for another machine
+zen meta dataset export -o dataset.json
+```
+
+`export [id...]` is the same filters with no strata, in id order, as `cases`
+by default.
+
+## Rules
+
+- Only `zen meta dataset` writes `dataset/`.
+- Every write says `--why`, in terms of the source: what changed in it, or why
+  it is being overridden.
+- Rubrics leave the dataset only through `--format cases` or `export`. Never
+  put one in a batch input, a prompt, a skill or the workspace an agent reads.
+- Moving a rubric to match what an agent did is not a fix; it deletes the test.