@sentry/warden 0.22.0 → 0.24.0

package/src/internal-skills/skill-writer/references/examples/security-review-skill.md

@@ -0,0 +1,69 @@
+# Case Study: Security Review Skill Synthesis
+
+## Scenario
+
+Goal: build a skill that finds real vulnerabilities while minimizing false positives.
+
+## Input collection approach
+
+This case required balanced collection across offensive and defensive material:
+
+1. Canonical standards and cheat sheets.
+2. Framework/language-specific secure coding docs.
+3. Real-world exploit writeups and postmortems.
+4. Fixed vulnerability diffs and secure rewrites.
+5. Benign patterns often misclassified as vulnerabilities.
+6. Existing in-repo security skills and review heuristics.
+
+Collection continued until each vulnerability class had both exploit and mitigation evidence.
+
+## Coverage matrix used
+
+Required dimensions tracked during synthesis:
+
+1. Vulnerability class definitions and prerequisites.
+2. Exploitable dataflow examples.
+3. False-positive controls.
+4. Severity/confidence calibration.
+5. Concrete remediation patterns.
+6. Framework-specific caveats and exceptions.
+
+## Synthesized artifacts produced
+
+The resulting skill references included:
+
+1. True-positive case with exploitation path.
+2. False-positive case with proof of safety.
+3. Fix/remediation case with corrected code pattern.
+4. Severity and confidence decision rubric.
+5. Evidence checklist to prevent pattern-only claims.
+
+## Source-to-decision trace (sample)
+
+1. Source class: exploit writeups.
+   Decision: require attacker-controlled input path in every high-confidence finding.
+   Why: removed pattern-only false alarms.
+2. Source class: benign counterexamples.
+   Decision: add explicit safe-pattern checks before reporting.
+   Why: reduced repeated false positives on sanitized data paths.
+3. Source class: fixed vulnerability diffs.
+   Decision: include remediation examples as patch-shaped guidance.
+   Why: improved downstream fix quality and speed.
+
+## Concrete artifacts (sample)
+
+1. True-positive case:
+   Input pattern: untrusted data reaches shell/API call without escaping.
+   Output: finding includes source, sink, exploit path, and minimal patch recommendation.
+2. False-positive case:
+   Input pattern: potentially dangerous API with validated allowlist and strict escaping.
+   Output: no vulnerability finding; include reason for non-reporting.
+3. Remediation case:
+   Before: dynamic query construction from user input.
+   After: parameterized query plus validation guard.
+
+## What made this high quality
+
+1. It was trained on both attacks and safe counterexamples.
+2. Findings required evidence of exploitability, not keyword matching.
+3. Remediation guidance was concrete and immediately applicable.

package/src/internal-skills/skill-writer/references/examples/subagent-fork-skill.md

@@ -0,0 +1,39 @@
+# Case Study: Subagent-Fork Skill Synthesis
+
+## Scenario
+
+Goal: create a skill that runs a self-contained task in isolated context and returns a concise summary.
+
+## Input collection approach
+
+This case collected:
+
+1. official provider docs for skill execution in forked context
+2. examples of self-contained tasks that benefit from isolation
+3. failure cases where passive guidance was incorrectly put into subagents
+4. summary expectations for returning results to the main thread
+
+## Coverage matrix used
+
+Required dimensions tracked during synthesis:
+
+1. why isolation helps
+2. task prompt clarity
+3. output/summary contract
+4. tool/model assumptions
+5. portability constraints
+
+## Synthesized artifacts produced
+
+The resulting skill references included:
+
+1. actionable task body
+2. expected summary schema
+3. portability note
+4. anti-pattern showing passive guidance that should stay inline
+
+## What made this high quality
+
+1. the skill body was a task, not a convention list
+2. the isolation benefit was concrete
+3. the result expected back in the main thread was explicit

package/src/internal-skills/skill-writer/references/examples/workflow-process-skill.md

@@ -0,0 +1,65 @@
+# Case Study: Workflow/Process Skill Synthesis
+
+## Scenario
+
+Goal: create a skill for repeatable operational workflows (for example PR prep, CI triage, branching, settings audit).
+
+## Input collection approach
+
+This case collected process truth from all authoritative locations:
+
+1. Official tool docs and syntax references.
+2. Repository workflow conventions and policy docs.
+3. Existing local skills with adjacent process logic.
+4. CI logs, failure patterns, and known operational pitfalls.
+5. Positive and negative historical examples from prior runs.
+
+Collection stopped only after failure and recovery paths were well represented.
+
+## Coverage matrix used
+
+Required dimensions tracked during synthesis:
+
+1. Preconditions and required context.
+2. Ordered execution flow.
+3. Safety/permission boundaries.
+4. Expected outputs and acceptance checks.
+5. Failure handling and retry behavior.
+6. Escalation and handoff behavior.
+
+## Synthesized artifacts produced
+
+The resulting skill references included:
+
+1. Happy-path execution transcript.
+2. Guarded variant with stricter safety constraints.
+3. Failure-recovery transcript for a critical broken step.
+4. Output template for deterministic reporting.
+5. Changelog rules for iterative improvement from examples.
+
+## Source-to-decision trace (sample)
+
+1. Source class: repo policy docs.
+   Decision: add explicit precondition checks before running side-effecting steps.
+   Why: prevented invalid execution in partially configured environments.
+2. Source class: CI failure logs.
+   Decision: add a mandatory failure triage branch with retry vs escalate criteria.
+   Why: reduced dead-end loops during workflow execution.
+3. Source class: historical positive/negative examples.
+   Decision: standardize output format for easier review and iteration.
+   Why: made regressions and improvements comparable across runs.
+
+## Concrete artifacts (sample)
+
+1. Happy-path transcript snippet:
+   Preconditions pass -> execute steps 1..N -> emit structured summary with status per step.
+2. Failure-recovery transcript snippet:
+   Step fails -> classify transient/permanent -> retry once or escalate with captured evidence.
+3. Deterministic report template:
+   Sections: Preconditions, Actions Taken, Validation Results, Failures/Recoveries, Next Actions.
+
+## What made this high quality
+
+1. The workflow was executable without rediscovering steps.
+2. Non-happy paths were first-class, not afterthoughts.
+3. Outputs were structured for consistent review and iteration.

package/src/internal-skills/skill-writer/references/execution-shapes.md

@@ -0,0 +1,68 @@
+# Execution Shapes
+
+Use this guide to choose the runtime shape of a skill before you decide its files.
+Default rule: choose the simplest adequate shape, then add complexity only when it clearly improves outcomes.
+Once you pick a shape, load only the concrete leaf references it needs.
+
+## Defaulting To The Simplest Shape
+
+Start from these questions, in order:
+
+1. Can one coherent set of instructions handle most requests?
+   If yes, prefer `inline-guidance`.
+2. Is the main complexity optional knowledge rather than control flow?
+   If yes, prefer `reference-backed-expert`.
+3. Is the hard part data extraction, validation, or repeatable automation?
+   If yes, prefer `script-backed-workflow`.
+4. Does the user usually invoke the skill with explicit parameters?
+   If yes, add `argument-driven`.
+5. Only then consider routing, worker delegation, evaluator loops, subagent execution, hooks, or templates.
+
+Do not jump to advanced mechanics because they sound powerful.
+
+## Shape Decision Table
+
+| Shape | Use when | Open next | Portability notes |
+|-------|----------|-----------|-------------------|
+| `inline-guidance` | one coherent policy, checklist, or procedure is enough | `references/artifact-layouts/inline-skill-layout.md` | most portable default |
+| `reference-backed-expert` | optional deep knowledge is the main complexity | `references/artifact-layouts/reference-backed-skill-layout.md` | portable if file references stay relative |
+| `script-backed-workflow` | repeated parsing, validation, APIs, or transformations are fragile in plain shell | `references/artifact-layouts/script-backed-skill-layout.md` | portable if dependencies are explicit |
+| `argument-driven` | the skill is usually invoked with issue numbers, paths, targets, or modes | `references/artifact-layouts/argument-driven-skill-layout.md` | often provider-specific beyond basic manual invocation |
+| `router` | distinct categories need different downstream prompts, tools, or references | `references/workflow-mechanics/routing-workflows.md` | portable if routing stays in prompt logic |
+| `parallelization` | independent subtasks or multiple votes improve speed or confidence | `references/workflow-mechanics/parallel-workflows.md` | often implemented with tools or agents |
+| `orchestrator-workers` | the number or type of subtasks is discovered at runtime | `references/workflow-mechanics/orchestrator-workers.md` | usually higher-latency and provider-sensitive |
+| `evaluator-optimizer` | critique-and-revise loops improve output quality materially | `references/workflow-mechanics/evaluator-loops.md` | portable in concept; costly if overused |
+| `subagent-fork` | the skill needs isolated context, tools, or model defaults | `references/claude-code/subagent-fork-skills.md` | Claude Code-specific |
+| `hook-backed` | deterministic enforcement is required beyond prompt guidance | `references/claude-code/hook-backed-skills.md` | highly provider-specific and security-sensitive |
+| `asset-template` | reusable templates, schemas, or static artifacts carry most of the value | `references/artifact-layouts/asset-template-skill-layout.md` | portable if assets are generic files |
+
+If the chosen shape also uses explicit arguments, Claude-specific frontmatter, or shell preprocessing, load the matching file from `references/claude-code/`.
+
+## Secondary Workflow Mechanics
+
+These are not usually primary execution shapes, but they often refine one:
+
+- fixed ordered steps -> `references/workflow-mechanics/prompt-chaining.md`
+- validate-fix-repeat loops -> `references/workflow-mechanics/validation-loops.md`
+- plan-before-execute flows -> `references/workflow-mechanics/plan-validate-execute.md`
+
+## Hybrid Shapes
+
+Use a hybrid only when one primary shape is insufficient.
+
+1. Declare one primary shape.
+2. Add only the minimum secondary shapes needed.
+3. Keep each secondary shape scoped to one concrete need.
+4. Avoid stacking multiple advanced shapes without a clear base path.
+
+## Advanced-Shape Hard Stops
+
+Do not finalize a skill when any of these are true:
+
+1. The chosen shape is implied but not named.
+2. A simpler shape was not considered.
+3. A router has no fallback or default path.
+4. An evaluator loop has no stopping rule.
+5. A subagent-fork skill contains only passive guidance.
+6. A hook-backed skill lacks a security note or fallback behavior.
+7. Provider-specific mechanics are used without portability notes.

package/src/internal-skills/skill-writer/references/iteration-evidence.md

@@ -0,0 +1,82 @@
+# Iteration Evidence
+
+Use this guide when improving a skill from positive examples, negative examples, review feedback, eval results, or observed agent behavior.
+
+## Storage Layout
+
+Store persistent improvement evidence under:
+
+```text
+references/evidence/
+├── findings-log.md
+├── working-set.md
+└── holdout-set.md
+```
+
+Use this directory only when examples should outlive the current task. For a one-off small fix, summarize the examples in `SOURCES.md` instead.
+
+## File Roles
+
+`references/evidence/findings-log.md` records interpreted findings:
+
+- repeated failure patterns
+- preserved success patterns
+- suspected root causes
+- instruction changes made in response
+- unresolved risks
+
+`references/evidence/working-set.md` stores examples used while editing the skill.
+
+`references/evidence/holdout-set.md` stores examples reserved for validation after edits. Do not tune directly against holdout examples unless the user explicitly moves them into the working set.
+
+## Example Record Schema
+
+Use one record per example:
+
+```markdown
+## EX-001: Short label
+
+- Label: positive | negative
+- Kind: true-positive | false-positive | false-negative | fix | regression | edge-case
+- Origin: human-verified | mixed | synthetic
+- Source: issue/PR/commit/log/user note/local eval pointer
+- Status: working | holdout | resolved | deferred
+- Expected behavior: concise statement
+- Observed behavior: concise statement
+- Skill delta: instruction, reference, description, or eval change
+- Anonymization: what was removed or generalized
+
+### Content
+
+Summarized or redacted example content.
+```
+
+Keep records concise. Preserve enough detail to reproduce the behavior, but redact secrets, customer data, private URLs, and unnecessary user content.
+
+## Positive And Negative Findings
+
+Positive findings are not just success stories. Use them to protect behaviors that must not regress.
+
+Negative findings should identify the smallest failing decision:
+
+- wrong trigger behavior
+- missing source type
+- skipped reference file
+- overloaded or hidden instruction
+- weak output contract
+- missing validation step
+- unsafe or non-portable path assumption
+
+Each negative finding should map to a concrete skill delta or an explicit deferred reason.
+
+## Promotion Rules
+
+Promote evidence into the skill artifacts only when it changes future behavior:
+
+- Put universal behavioral rules in `SKILL.md`.
+- Put domain-specific examples in a focused reference.
+- Put source provenance and decisions in `SOURCES.md`.
+- Put reusable eval prompts in `EVAL.md`.
+- Keep raw or semi-raw examples in `references/evidence/`.
+
+Do not turn `references/evidence/` into a changelog. The changelog belongs in `SOURCES.md`.

package/src/internal-skills/skill-writer/references/iteration-path.md

@@ -0,0 +1,40 @@
+# Iteration Path
+
+Use this path when improving a skill based on outcomes and examples.
+
+## Example intake
+
+Read `references/iteration-evidence.md` when examples should be persisted across future skill revisions.
+
+Capture example records with:
+
+- label (`positive` or `negative`)
+- example kind (`true-positive`, `false-positive`, `fix`, `regression`, `edge-case`)
+- evidence origin (`human-verified`, `mixed`, `synthetic`)
+- anonymized content
+- source provenance pointer (where the example came from)
+
+## Replay and evaluation
+
+1. Evaluate against working set.
+2. Evaluate against holdout set.
+3. Record improved/unchanged/regressed outcomes.
+4. Confirm both positive and negative behavior changed in the expected direction.
+
+## Improvement rules
+
+1. Prioritize fixes for repeated negative patterns.
+2. Preserve behavior that consistently succeeds on positives.
+3. Update transformed examples when guidance changes.
+4. Record deltas in `SOURCES.md` changelog.
+5. Expand input collection when failures indicate coverage gaps.
+6. Store durable positive/negative examples in `references/evidence/` instead of overloading `SKILL.md`, `SOURCES.md`, or a generic reference file.
+7. Keep holdout examples separate from working examples until validation is complete.
+8. Update `SPEC.md` when iteration changes the skill's intended scope, evidence model, evaluation gates, or known limitations.
+
+## Required output
+
+- Example intake summary
+- Behavior deltas
+- Updated artifacts
+- Replay summary

package/src/internal-skills/skill-writer/references/mode-selection.md

@@ -0,0 +1,67 @@
+# Mode Selection
+
+Choose the minimal set of paths needed for the request.
+Regardless of path, prioritize input quality and coverage depth before finalizing outputs.
+Do not add evaluation unless the user asks for it, the change is high-risk, or the architecture choice is genuinely uncertain.
+
+## Path mapping
+
+| Request shape | Required paths |
+|---------------|----------------|
+| New skill from scratch | synthesis + authoring + description optimization + registration/validation |
+| Update existing skill wording/structure | authoring + description optimization + registration/validation |
+| Improve skill from outcomes/examples | iteration + authoring + description optimization + registration/validation |
+| Research-first skill planning | synthesis only, then authoring if requested |
+| Risky, disputed, or explicitly reviewed change | add `evaluation` to the selected path |
+
+## Skill class selection
+
+Classify the target skill before synthesis. This determines the coverage dimensions that must be represented in sources, references, and validation.
+
+| Skill class | Typical request shape | Required dimensions |
+|-------------|-----------------------|---------------------|
+| `workflow-process` | repeatable operations, CI/task orchestration | preconditions, ordered flow, failure handling, safety boundaries |
+| `integration-documentation` | library/framework integration, SDK usage, API correctness | API surface, config/runtime options, common use cases, known issues/workarounds, version/migration variance |
+| `security-review` | vulnerability finding, exploitability review | vulnerability classes, exploit paths, false-positive controls, remediations |
+| `skill-authoring` | creating/updating/evaluating other skills | source provenance, depth gates, transformed examples, registration/validation |
+| `generic` | does not match above | explicit dimensions chosen and justified in synthesis |
+
+When the class is ambiguous, ask one direct clarification question before synthesis.
+
+## Execution shape selection
+
+Choose the skill's primary execution shape separately from the skill class.
+Class answers "what domain/problem is this skill for?"
+Shape answers "how should this skill run?"
+
+Use `references/execution-shapes.md` for the full decision table and the next leaf reference to load.
+
+Record:
+
+1. primary execution shape
+2. simpler-shape rejection when the chosen shape is advanced
+3. exact leaf references opened because of that choice
+
+## Required outputs by path
+
+- `synthesis`: source inventory, decisions, coverage matrix, gaps.
+- `synthesis`: selected class, selected execution shape, and selected example profile path(s), including profile-requirement coverage.
+- `synthesis`: simplicity rationale showing why the chosen shape is necessary and which simpler shapes were rejected.
+- `synthesis`: portability note when provider-specific mechanics are used.
+- `synthesis`: explicit retrieval stopping rationale showing why further collection is currently low-yield.
+- `authoring`: updated `SKILL.md` and required supporting files.
+- `description optimization`: should/should-not trigger sets and final description.
+- `iteration`: example intake summary and behavior deltas.
+- `evaluation`: qualitative summary and any deeper checks run.
+- `registration/validation`: registration edits and validator results.
+
+## Hard stop rules
+
+Do not claim completion when any required path output is missing.
+Evaluation output is required only when `evaluation` was selected.
+
+For authoring/generator skills, missing transformed example artifacts is a hard failure.
+Missing selected-profile requirements is also a hard failure.
+Missing required class dimensions is a hard failure.
+Missing an explicit execution-shape choice for a material skill change is a hard failure.
+Using advanced mechanics without justification or portability notes is a hard failure.

package/src/internal-skills/skill-writer/references/output-contracts.md

@@ -0,0 +1,68 @@
+# Output Contracts
+
+Use this guide when the skill needs a predictable response shape.
+
+## Choose The Contract
+
+| Need | Use |
+|------|-----|
+| exact sections or headings | strict template |
+| default structure with adaptation | flexible template |
+| style is easier to imitate than describe | input/output examples |
+| format depends on task type | decision table |
+| scripts or tools parse the output | structured schema |
+
+## Strict Template
+
+```markdown
+# [Title]
+
+## Summary
+[Required summary]
+
+## Findings
+- ...
+```
+
+## Flexible Template
+
+```markdown
+# [Title]
+
+## Summary
+[Default summary section; adapt if needed]
+
+## Findings
+[Adapt based on context]
+```
+
+## Input/Output Example
+
+````markdown
+Input: fix date formatting bug
+Output:
+```text
+fix(reports): correct timezone date formatting
+```
+````
+
+## Decision Table
+
+```markdown
+| Input Type | Output Format |
+|------------|---------------|
+| single file | inline summary |
+| many files | grouped report |
+```
+
+## Structured Schema
+
+````markdown
+```json
+{
+  "status": "success",
+  "summary": "One-line result",
+  "findings": []
+}
+```
+````

package/src/internal-skills/skill-writer/references/reference-architecture.md

@@ -0,0 +1,71 @@
+# Reference Architecture
+
+Use this guide before adding bundled files or long sections to `SKILL.md`.
+
+## Router Rule
+
+- `SKILL.md` is the router.
+- References are lookup leaves.
+- `SPEC.md` is the maintenance contract.
+- `SOURCES.md` stores provenance and decisions.
+- `EVAL.md` stores reusable eval prompts or runbooks.
+
+## Lookup Test
+
+Before creating a reference, finish this sentence:
+
+- "I need to decide X, so read `...`."
+- "I need to do Y, so read `...`."
+- "I need to diagnose Z, so read `...`."
+
+If the sentence sounds like "I need context" or "I need patterns", the file is too vague.
+
+## Placement Table
+
+| Put it in... | When it belongs there |
+|--------------|-----------------------|
+| `SKILL.md` | every run needs it |
+| `references/` | only some branches need it |
+| `SPEC.md` | it explains maintenance, scope, or evidence policy |
+| `SOURCES.md` | it is provenance, a decision record, or a gap |
+| `EVAL.md` | it is a reusable eval prompt or grading runbook |
+
+## Reference Types
+
+| Need | Shape |
+|------|-------|
+| choose a path | decision guide |
+| execute a procedure | task guide |
+| look up exact facts | reference table |
+| diagnose a failure | troubleshooting matrix |
+| imitate quality | example set |
+| judge completeness | evaluation rubric |
+
+## Naming Rules
+
+- Name files for the question or action they answer.
+- Good: `mode-selection.md`, `output-contracts.md`, `routing-workflows.md`
+- Bad: `notes.md`, `context.md`, `patterns.md`, `research.md`
+- Use subfolders only when the subtree name clarifies the lookup path.
+
+## Split Rules
+
+Create a new reference when:
+
+- the content is only needed after a branch decision
+- the content has one dominant type
+- the section would make `SKILL.md` harder to scan
+- the file is approaching 100 lines and contains multiple lookup needs
+
+Keep content in `SKILL.md` when:
+
+- every invocation needs it
+- it is short
+- moving it would force unnecessary file loads
+
+## Final Checks
+
+1. Every runtime reference has a direct "open when..." reason in `SKILL.md`.
+2. The filename tells the agent why to open it.
+3. No required instruction is hidden only in an optional reference.
+4. No reference has become a second `SKILL.md`.

package/src/internal-skills/skill-writer/references/registration-validation.md

@@ -0,0 +1,67 @@
+# Registration and Validation
+
+Apply registration and quality checks before completion.
+
+## Registration checklist
+
+1. Inspect the workspace and identify the active skill layout before editing files.
+2. Create/update `<skill-root>/SKILL.md`, `<skill-root>/SPEC.md` when required by change scope, and any bundled `references/`, `scripts/`, or `assets/` beneath that root.
+3. Default to `.agents/skills/<name>/` when there is no stronger prior art.
+4. If the workspace clearly uses a different canonical layout, follow that layout instead of forcing `.agents/skills/`.
+5. Common established alternatives include:
+   - `skills/<name>/` when the workspace uses a canonical root skill tree
+   - `.claude/skills/<name>/` for project-scoped Claude skills
+   - `plugins/<plugin>/skills/<name>/` for plugin-scoped skills
+   - another repository-managed skill root that is already established by neighboring skills or docs
+6. If multiple plausible locations exist and inspection does not make the canonical target clear, ask the user before editing files.
+7. Only apply repository-specific registration steps when the workspace conventions explicitly require them.
+
+When a repository does maintain its own skill catalog, verify and update any required registration files such as:
+
+- public skill inventories or tables
+- project or plugin settings files
+- allowlists used by other skills or automation
+
+## Validation checklist
+
+1. Run:
+
+```bash
+uv run scripts/quick_validate.py <path/to/skill-directory> --strict-depth
+```
+
+Use the skill-root-relative form above when running from the `skill-writer` directory.
+If you must run the validator from another working directory, convert both paths to the correct relative path from that directory instead of introducing absolute or host-specific paths into the skill docs.
+
+2. Confirm for authoring/generator skills:
+- transformed examples exist in references (happy-path, secure/robust, anti-pattern+fix)
+- synthesis depth gates are satisfied
+- selected example profile requirements are satisfied and reported
+- `SPEC.md` exists or was updated when the change creates a skill or materially changes intent, scope, evidence model, evaluation, or maintenance expectations
+- every bundled reference file is directly discoverable from `SKILL.md`
+
+3. Confirm for integration/documentation skills:
+- focused references cover API surface, common use cases, known issues/workarounds, and version variance
+- reference file names fit the skill's domain rather than a fixed template
+- `SKILL.md` and `references/*.md` avoid host-specific absolute filesystem paths
+
+4. Confirm portability for skills that are expected to be portable by default:
+- bundled file references use skill-root-relative paths such as `references/...`, `scripts/...`, or `assets/...`
+- provider-specific path variables (for example `${CLAUDE_SKILL_ROOT}`) either follow established repository prior art or are explicitly scoped, rather than being introduced ad hoc
+- provider-specific behavior, if any, is labeled as compatibility guidance rather than the primary workflow
+
+5. If evaluation was run, include:
+- lightweight qualitative summary
+- concise depth rubric status for integration/documentation and skill-authoring skills
+- deeper eval or quantitative summary only if user requested it or risk warranted it
+
+6. Review validator warnings for long reference files and split files when warnings indicate mixed concerns or poor navigation.
+7. Reject shallow handoffs that omit required artifacts.
+
+## Required output
+
+- Registration changes summary
+- Selected skill root and why it was chosen
+- Validator output
+- Evaluation summary, if run
+- Any residual risks or open gaps