@sentry/warden 0.21.0 → 0.23.0

package/src/internal-skills/skill-writer/references/iteration-evidence.md

@@ -0,0 +1,82 @@
+# Iteration Evidence
+
+Use this guide when improving a skill from positive examples, negative examples, review feedback, eval results, or observed agent behavior.
+
+## Storage Layout
+
+Store persistent improvement evidence under:
+
+```text
+references/evidence/
+├── findings-log.md
+├── working-set.md
+└── holdout-set.md
+```
+
+Use this directory only when examples should outlive the current task. For a one-off small fix, summarize the examples in `SOURCES.md` instead.
+
+## File Roles
+
+`references/evidence/findings-log.md` records interpreted findings:
+
+- repeated failure patterns
+- preserved success patterns
+- suspected root causes
+- instruction changes made in response
+- unresolved risks
+
+`references/evidence/working-set.md` stores examples used while editing the skill.
+
+`references/evidence/holdout-set.md` stores examples reserved for validation after edits. Do not tune directly against holdout examples unless the user explicitly moves them into the working set.
+
+## Example Record Schema
+
+Use one record per example:
+
+```markdown
+## EX-001: Short label
+
+- Label: positive | negative
+- Kind: true-positive | false-positive | false-negative | fix | regression | edge-case
+- Origin: human-verified | mixed | synthetic
+- Source: issue/PR/commit/log/user note/local eval pointer
+- Status: working | holdout | resolved | deferred
+- Expected behavior: concise statement
+- Observed behavior: concise statement
+- Skill delta: instruction, reference, description, or eval change
+- Anonymization: what was removed or generalized
+
+### Content
+
+Summarized or redacted example content.
+```
+
+Keep records concise. Preserve enough detail to reproduce the behavior, but redact secrets, customer data, private URLs, and unnecessary user content.
+
+## Positive And Negative Findings
+
+Positive findings are not just success stories. Use them to protect behaviors that must not regress.
+
+Negative findings should identify the smallest failing decision:
+
+- wrong trigger behavior
+- missing source type
+- skipped reference file
+- overloaded or hidden instruction
+- weak output contract
+- missing validation step
+- unsafe or non-portable path assumption
+
+Each negative finding should map to a concrete skill delta or an explicit deferred reason.
+
+## Promotion Rules
+
+Promote evidence into the skill artifacts only when it changes future behavior:
+
+- Put universal behavioral rules in `SKILL.md`.
+- Put domain-specific examples in a focused reference.
+- Put source provenance and decisions in `SOURCES.md`.
+- Put reusable eval prompts in `EVAL.md`.
+- Keep raw or semi-raw examples in `references/evidence/`.
+
+Do not turn `references/evidence/` into a changelog. The changelog belongs in `SOURCES.md`.

package/src/internal-skills/skill-writer/references/iteration-path.md

@@ -0,0 +1,40 @@
+# Iteration Path
+
+Use this path when improving a skill based on outcomes and examples.
+
+## Example intake
+
+Read `references/iteration-evidence.md` when examples should be persisted across future skill revisions.
+
+Capture example records with:
+
+- label (`positive` or `negative`)
+- example kind (`true-positive`, `false-positive`, `fix`, `regression`, `edge-case`)
+- evidence origin (`human-verified`, `mixed`, `synthetic`)
+- anonymized content
+- source provenance pointer (where the example came from)
+
+## Replay and evaluation
+
+1. Evaluate against working set.
+2. Evaluate against holdout set.
+3. Record improved/unchanged/regressed outcomes.
+4. Confirm both positive and negative behavior changed in the expected direction.
+
+## Improvement rules
+
+1. Prioritize fixes for repeated negative patterns.
+2. Preserve behavior that consistently succeeds on positives.
+3. Update transformed examples when guidance changes.
+4. Record deltas in `SOURCES.md` changelog.
+5. Expand input collection when failures indicate coverage gaps.
+6. Store durable positive/negative examples in `references/evidence/` instead of overloading `SKILL.md`, `SOURCES.md`, or a generic reference file.
+7. Keep holdout examples separate from working examples until validation is complete.
+8. Update `SPEC.md` when iteration changes the skill's intended scope, evidence model, evaluation gates, or known limitations.
+
+## Required output
+
+- Example intake summary
+- Behavior deltas
+- Updated artifacts
+- Replay summary

package/src/internal-skills/skill-writer/references/mode-selection.md

@@ -0,0 +1,67 @@
+# Mode Selection
+
+Choose the minimal set of paths needed for the request.
+Regardless of path, prioritize input quality and coverage depth before finalizing outputs.
+Do not add evaluation unless the user asks for it, the change is high-risk, or the architecture choice is genuinely uncertain.
+
+## Path mapping
+
+| Request shape | Required paths |
+|---------------|----------------|
+| New skill from scratch | synthesis + authoring + description optimization + registration/validation |
+| Update existing skill wording/structure | authoring + description optimization + registration/validation |
+| Improve skill from outcomes/examples | iteration + authoring + description optimization + registration/validation |
+| Research-first skill planning | synthesis only, then authoring if requested |
+| Risky, disputed, or explicitly reviewed change | add `evaluation` to the selected path |
+
+## Skill class selection
+
+Classify the target skill before synthesis. This determines the coverage dimensions that must be represented in sources, references, and validation.
+
+| Skill class | Typical request shape | Required dimensions |
+|-------------|-----------------------|---------------------|
+| `workflow-process` | repeatable operations, CI/task orchestration | preconditions, ordered flow, failure handling, safety boundaries |
+| `integration-documentation` | library/framework integration, SDK usage, API correctness | API surface, config/runtime options, common use cases, known issues/workarounds, version/migration variance |
+| `security-review` | vulnerability finding, exploitability review | vulnerability classes, exploit paths, false-positive controls, remediations |
+| `skill-authoring` | creating/updating/evaluating other skills | source provenance, depth gates, transformed examples, registration/validation |
+| `generic` | does not match above | explicit dimensions chosen and justified in synthesis |
+
+When the class is ambiguous, ask one direct clarification question before synthesis.
+
+## Execution shape selection
+
+Choose the skill's primary execution shape separately from the skill class.
+Class answers "what domain/problem is this skill for?"
+Shape answers "how should this skill run?"
+
+Use `references/execution-shapes.md` for the full decision table and the next leaf reference to load.
+
+Record:
+
+1. primary execution shape
+2. simpler-shape rejection when the chosen shape is advanced
+3. exact leaf references opened because of that choice
+
+## Required outputs by path
+
+- `synthesis`: source inventory, decisions, coverage matrix, gaps.
+- `synthesis`: selected class, selected execution shape, and selected example profile path(s), including profile-requirement coverage.
+- `synthesis`: simplicity rationale showing why the chosen shape is necessary and which simpler shapes were rejected.
+- `synthesis`: portability note when provider-specific mechanics are used.
+- `synthesis`: explicit retrieval stopping rationale showing why further collection is currently low-yield.
+- `authoring`: updated `SKILL.md` and required supporting files.
+- `description optimization`: should/should-not trigger sets and final description.
+- `iteration`: example intake summary and behavior deltas.
+- `evaluation`: qualitative summary and any deeper checks run.
+- `registration/validation`: registration edits and validator results.
+
+## Hard stop rules
+
+Do not claim completion when any required path output is missing.
+Evaluation output is required only when `evaluation` was selected.
+
+For authoring/generator skills, missing transformed example artifacts is a hard failure.
+Missing selected-profile requirements is also a hard failure.
+Missing required class dimensions is a hard failure.
+Missing an explicit execution-shape choice for a material skill change is a hard failure.
+Using advanced mechanics without justification or portability notes is a hard failure.

package/src/internal-skills/skill-writer/references/output-contracts.md

@@ -0,0 +1,68 @@
+# Output Contracts
+
+Use this guide when the skill needs a predictable response shape.
+
+## Choose The Contract
+
+| Need | Use |
+|------|-----|
+| exact sections or headings | strict template |
+| default structure with adaptation | flexible template |
+| style is easier to imitate than describe | input/output examples |
+| format depends on task type | decision table |
+| scripts or tools parse the output | structured schema |
+
+## Strict Template
+
+```markdown
+# [Title]
+
+## Summary
+[Required summary]
+
+## Findings
+- ...
+```
+
+## Flexible Template
+
+```markdown
+# [Title]
+
+## Summary
+[Default summary section; adapt if needed]
+
+## Findings
+[Adapt based on context]
+```
+
+## Input/Output Example
+
+````markdown
+Input: fix date formatting bug
+Output:
+```text
+fix(reports): correct timezone date formatting
+```
+````
+
+## Decision Table
+
+```markdown
+| Input Type | Output Format |
+|------------|---------------|
+| single file | inline summary |
+| many files | grouped report |
+```
+
+## Structured Schema
+
+````markdown
+```json
+{
+  "status": "success",
+  "summary": "One-line result",
+  "findings": []
+}
+```
+````

package/src/internal-skills/skill-writer/references/reference-architecture.md

@@ -0,0 +1,71 @@
+# Reference Architecture
+
+Use this guide before adding bundled files or long sections to `SKILL.md`.
+
+## Router Rule
+
+- `SKILL.md` is the router.
+- References are lookup leaves.
+- `SPEC.md` is the maintenance contract.
+- `SOURCES.md` stores provenance and decisions.
+- `EVAL.md` stores reusable eval prompts or runbooks.
+
+## Lookup Test
+
+Before creating a reference, finish this sentence:
+
+- "I need to decide X, so read `...`."
+- "I need to do Y, so read `...`."
+- "I need to diagnose Z, so read `...`."
+
+If the sentence sounds like "I need context" or "I need patterns", the file is too vague.
+
+## Placement Table
+
+| Put it in... | When it belongs there |
+|--------------|-----------------------|
+| `SKILL.md` | every run needs it |
+| `references/` | only some branches need it |
+| `SPEC.md` | it explains maintenance, scope, or evidence policy |
+| `SOURCES.md` | it is provenance, a decision record, or a gap |
+| `EVAL.md` | it is a reusable eval prompt or grading runbook |
+
+## Reference Types
+
+| Need | Shape |
+|------|-------|
+| choose a path | decision guide |
+| execute a procedure | task guide |
+| look up exact facts | reference table |
+| diagnose a failure | troubleshooting matrix |
+| imitate quality | example set |
+| judge completeness | evaluation rubric |
+
+## Naming Rules
+
+- Name files for the question or action they answer.
+- Good: `mode-selection.md`, `output-contracts.md`, `routing-workflows.md`
+- Bad: `notes.md`, `context.md`, `patterns.md`, `research.md`
+- Use subfolders only when the subtree name clarifies the lookup path.
+
+## Split Rules
+
+Create a new reference when:
+
+- the content is only needed after a branch decision
+- the content has one dominant type
+- the section would make `SKILL.md` harder to scan
+- the file is approaching 100 lines and contains multiple lookup needs
+
+Keep content in `SKILL.md` when:
+
+- every invocation needs it
+- it is short
+- moving it would force unnecessary file loads
+
+## Final Checks
+
+1. Every runtime reference has a direct "open when..." reason in `SKILL.md`.
+2. The filename tells the agent why to open it.
+3. No required instruction is hidden only in an optional reference.
+4. No reference has become a second `SKILL.md`.

package/src/internal-skills/skill-writer/references/registration-validation.md

@@ -0,0 +1,67 @@
+# Registration and Validation
+
+Apply registration and quality checks before completion.
+
+## Registration checklist
+
+1. Inspect the workspace and identify the active skill layout before editing files.
+2. Create/update `<skill-root>/SKILL.md`, `<skill-root>/SPEC.md` when required by change scope, and any bundled `references/`, `scripts/`, or `assets/` beneath that root.
+3. Default to `.agents/skills/<name>/` when there is no stronger prior art.
+4. If the workspace clearly uses a different canonical layout, follow that layout instead of forcing `.agents/skills/`.
+5. Common established alternatives include:
+   - `skills/<name>/` when the workspace uses a canonical root skill tree
+   - `.claude/skills/<name>/` for project-scoped Claude skills
+   - `plugins/<plugin>/skills/<name>/` for plugin-scoped skills
+   - another repository-managed skill root that is already established by neighboring skills or docs
+6. If multiple plausible locations exist and inspection does not make the canonical target clear, ask the user before editing files.
+7. Only apply repository-specific registration steps when the workspace conventions explicitly require them.
+
+When a repository does maintain its own skill catalog, verify and update any required registration files such as:
+
+- public skill inventories or tables
+- project or plugin settings files
+- allowlists used by other skills or automation
+
+## Validation checklist
+
+1. Run:
+
+```bash
+uv run scripts/quick_validate.py <path/to/skill-directory> --strict-depth
+```
+
+Use the skill-root-relative form above when running from the `skill-writer` directory.
+If you must run the validator from another working directory, convert both paths to the correct relative path from that directory instead of introducing absolute or host-specific paths into the skill docs.
+
+2. Confirm for authoring/generator skills:
+- transformed examples exist in references (happy-path, secure/robust, anti-pattern+fix)
+- synthesis depth gates are satisfied
+- selected example profile requirements are satisfied and reported
+- `SPEC.md` exists or was updated when the change creates a skill or materially changes intent, scope, evidence model, evaluation, or maintenance expectations
+- every bundled reference file is directly discoverable from `SKILL.md`
+
+3. Confirm for integration/documentation skills:
+- focused references cover API surface, common use cases, known issues/workarounds, and version variance
+- reference file names fit the skill's domain rather than a fixed template
+- `SKILL.md` and `references/*.md` avoid host-specific absolute filesystem paths
+
+4. Confirm portability for skills that are expected to be portable by default:
+- bundled file references use skill-root-relative paths such as `references/...`, `scripts/...`, or `assets/...`
+- provider-specific path variables (for example `${CLAUDE_SKILL_ROOT}`) either follow established repository prior art or are explicitly scoped, rather than being introduced ad hoc
+- provider-specific behavior, if any, is labeled as compatibility guidance rather than the primary workflow
+
+5. If evaluation was run, include:
+- lightweight qualitative summary
+- concise depth rubric status for integration/documentation and skill-authoring skills
+- deeper eval or quantitative summary only if user requested it or risk warranted it
+
+6. Review validator warnings for long reference files and split files when warnings indicate mixed concerns or poor navigation.
+7. Reject shallow handoffs that omit required artifacts.
+
+## Required output
+
+- Registration changes summary
+- Selected skill root and why it was chosen
+- Validator output
+- Evaluation summary, if run
+- Any residual risks or open gaps

package/src/internal-skills/skill-writer/references/source-discovery.md

@@ -0,0 +1,60 @@
+# Source Discovery
+
+Use this guide during synthesis when obvious docs are shallow, stale, incomplete, or too polished to reveal real behavior.
+
+## Source Priority
+
+Prefer sources in this order:
+
+1. Local repository authority: `AGENTS.md`, `README.md`, `CONTRIBUTING.md`, manifests, validators, scripts, tests, and existing neighboring skills.
+2. Primary upstream material: official specifications, product docs, API references, release notes, changelogs, and source code.
+3. Operational evidence: issue threads, PR discussions, CI failures, incident notes, support patterns, and migration notes.
+4. Historical evidence: commit logs, blame, reverted changes, and changelog diffs.
+5. Secondary summaries: blog posts, tutorials, and community examples.
+
+Treat secondary and generated content as leads, not authority.
+
+## High-Signal Retrieval Passes
+
+Run the passes that match the skill's risk:
+
+- Core behavior: official docs, source exports, public interfaces, happy-path examples.
+- Edge behavior: tests, fixtures, error handling, retries, permissions, validation, and cleanup paths.
+- Negative behavior: bug fixes, reverted commits, issue reports, support threads, review comments, and skipped tests.
+- Usage behavior: in-repo callers, downstream examples, configuration samples, and migration guides.
+- Maintenance behavior: changelog entries, release notes, deprecations, and commits touching the same files repeatedly.
+
+## Commit Log Mining
+
+Use commit history when the task depends on lived behavior rather than only intended behavior.
+
+Good candidates:
+
+- Security, access-control, CI, deployment, or migration skills.
+- Skills for internal systems with sparse public docs.
+- Skills being improved after repeated failures.
+- Areas with many regressions, reversions, or subtle edge cases.
+
+Useful commands:
+
+```bash
+git log --oneline -- <path>
+git log --stat -- <path>
+git log -G '<behavior|symbol|error>' -- <path>
+git blame <path>
+```
+
+Capture findings as source records with commit SHA, date, affected path, observed behavior, and whether the finding is adopted, rejected, or deferred.
+
+Do not copy large commit messages or diffs into `SKILL.md`. Summarize the behavior and keep provenance in `SOURCES.md` or an evidence file.
+
+## Stop Conditions
+
+Stop collecting when:
+
+- Required coverage dimensions are complete or have explicit next retrieval actions.
+- New retrieval mostly repeats known facts.
+- Remaining unknowns are low impact or require access the agent does not have.
+- The source mix includes both intended behavior and observed behavior for high-risk workflows.
+
+Record the stopping rationale in `SOURCES.md`.

package/src/internal-skills/skill-writer/references/spec-template.md

@@ -0,0 +1,118 @@
+# SPEC.md Template
+
+Use this guide to create or update a root-level `SPEC.md`.
+
+## Use `SPEC.md` For
+
+- intent
+- scope
+- trigger context
+- evidence model
+- evaluation expectations
+- limitations
+- maintenance rules
+
+Do not put runtime instructions or full provenance tables here.
+
+## Update `SPEC.md` When
+
+- intent or scope changes
+- trigger strategy changes
+- evidence sources or storage policy changes
+- reference architecture changes
+- evaluation gates change
+- privacy, security, or data-handling assumptions change
+
+For tiny wording-only fixes, update `SOURCES.md` changelog instead.
+
+## Relationship To Other Files
+
+| File | Purpose |
+|------|---------|
+| `SKILL.md` | runtime activation and execution |
+| `SPEC.md` | maintenance contract |
+| `SOURCES.md` | source inventory, decisions, gaps, changelog |
+| `EVAL.md` | reusable eval prompts or runbooks |
+| `references/` | runtime-loadable depth |
+| `references/evidence/` | persistent iteration examples |
+
+## Template
+
+```markdown
+# <Skill Name> Specification
+
+## Intent
+
+<1-2 short paragraphs>
+
+## Scope
+
+In scope:
+- ...
+
+Out of scope:
+- ...
+
+## Users And Trigger Context
+
+- Primary users:
+- Common user requests:
+- Should not trigger for:
+
+## Runtime Contract
+
+- Required first actions:
+- Required outputs:
+- Non-negotiable constraints:
+- Expected bundled files loaded at runtime:
+
+## Source And Evidence Model
+
+Authoritative sources:
+- ...
+
+Useful improvement sources:
+- positive examples:
+- negative examples:
+- commit logs/changelogs:
+- issue or PR feedback:
+- eval results:
+
+Data that must not be stored:
+- secrets
+- customer data
+- private URLs or identifiers not needed for reproduction
+
+## Reference Architecture
+
+- `SKILL.md` contains:
+- `references/` contains:
+- `references/evidence/` contains:
+- `scripts/` contains:
+- `assets/` contains:
+
+## Evaluation
+
+- Lightweight validation:
+- Deeper evaluation:
+- Holdout examples:
+- Acceptance gates:
+
+## Known Limitations
+
+- ...
+
+## Maintenance Notes
+
+- When to update `SKILL.md`:
+- When to update `SOURCES.md`:
+- When to update `EVAL.md`:
+- When to update `references/evidence/`:
+```
+
+## Design Rules
+
+1. Keep `SPEC.md` concise.
+2. Link to `SOURCES.md` or refs instead of duplicating them.
+3. Keep raw examples in `references/evidence/`.
+4. Keep sensitive data redacted.

package/src/internal-skills/skill-writer/references/structure-troubleshooting.md

@@ -0,0 +1,111 @@
+# Structure Troubleshooting
+
+Load this when the skill layout is unclear, overloaded, or drifting away from focused routing.
+
+## Over-long SKILL.md
+
+Problem: `SKILL.md` exceeds 500 lines and becomes a second encyclopedia.
+
+Fix: extract detailed material into focused `references/` files and keep `SKILL.md` as the router.
+
+## Missing Trigger Keywords
+
+Problem: the description is too vague to match user language.
+
+Fix: include the phrases users actually say.
+
+## Trigger Info In Body Instead Of Description
+
+Problem: "when to use" guidance appears only in the body, after triggering has already happened.
+
+Fix: move trigger language into `description`.
+
+## Duplicating CLAUDE.md Or AGENTS.md
+
+Problem: the skill repeats repo-wide conventions instead of adding net-new value.
+
+Fix: reference existing project docs and keep the skill focused on domain-specific behavior.
+
+## Unconditional Reference Loading
+
+Problem: the skill tells the agent to read every reference up front.
+
+Fix: add a decision table so references load only when needed.
+
+## Large References Without Navigation
+
+Problem: long reference files are hard to preview and easy to misuse.
+
+Fix: add a table of contents or split by lookup need.
+
+## Extraneous Files
+
+Problem: the skill directory accumulates user-facing docs or miscellaneous notes that do not help runtime, evaluation, or maintenance.
+
+Fix: keep only `SKILL.md`, `SPEC.md`, `SOURCES.md`, `EVAL.md`, `references/`, `scripts/`, `assets/`, and `LICENSE` when needed.
+
+## Scripts Without Documentation
+
+Problem: `SKILL.md` names a script but does not document arguments, output, or fallback behavior.
+
+Fix: document the script interface and expected output shape in `SKILL.md`.
+
+## Hardcoded Paths
+
+Problem: the skill embeds host-specific or repo-hardcoded paths.
+
+Fix: use skill-root-relative paths or established portable placeholders.
+
+## First/Second Person Descriptions
+
+Problem: the description says "I can..." or "You can use this..."
+
+Fix: write in third person so skill discovery stays consistent.
+
+## Time-Sensitive Information
+
+Problem: the skill bakes in dates or transitional logic that will quietly rot.
+
+Fix: move legacy behavior into a clearly labeled deprecated section or remove it.
+
+## Advanced Mechanics Without Justification
+
+Problem: the skill uses routing, `context: fork`, hooks, or evaluator loops because they seem sophisticated.
+
+Fix: name the shape, explain why simpler shapes were rejected, and add portability/safety notes.
+
+## Router Without Fallback
+
+Problem: the skill has multiple downstream paths but no default route or clarification step.
+
+Fix: add a fallback branch that asks one clarifying question or picks a documented safe default.
+
+## Evaluator Loop Without Stop Condition
+
+Problem: the skill says "iterate until good" with no rubric or cap.
+
+Fix: add a rubric plus a max-loop or plateau rule.
+
+## Passive Guidance In Forked Context
+
+Problem: `context: fork` is used for conventions or reference material instead of a concrete delegated task.
+
+Fix: keep passive guidance inline; use forked execution only for actionable isolated tasks.
+
+## Hook-Backed Enforcement Without Safety Notes
+
+Problem: the skill uses hooks but does not explain event scope, fallback behavior, or shell risk.
+
+Fix: add hook event scope, fallback path, and explicit security notes.
+
+## Hidden Reference File
+
+Problem: a bundled reference exists, but `SKILL.md` never tells the agent when to open it.
+
+Fix: add the file to the main router with a one-line "open when..." reason, or remove/split the file if no clear reason exists.
+
+## Generic Bucket Reference
+
+Problem: a file groups unrelated techniques under a vague name such as "patterns", "notes", or "context".
+
+Fix: split the file by lookup need and rename each leaf so the filename predicts why it should be opened.