@sentry/warden 0.22.0 → 0.23.0

package/src/internal-skills/skill-writer/references/spec-template.md

@@ -0,0 +1,118 @@
+# SPEC.md Template
+
+Use this guide to create or update a root-level `SPEC.md`.
+
+## Use `SPEC.md` For
+
+- intent
+- scope
+- trigger context
+- evidence model
+- evaluation expectations
+- limitations
+- maintenance rules
+
+Do not put runtime instructions or full provenance tables here.
+
+## Update `SPEC.md` When
+
+- intent or scope changes
+- trigger strategy changes
+- evidence sources or storage policy changes
+- reference architecture changes
+- evaluation gates change
+- privacy, security, or data-handling assumptions change
+
+For tiny wording-only fixes, update `SOURCES.md` changelog instead.
+
+## Relationship To Other Files
+
+| File | Purpose |
+|------|---------|
+| `SKILL.md` | runtime activation and execution |
+| `SPEC.md` | maintenance contract |
+| `SOURCES.md` | source inventory, decisions, gaps, changelog |
+| `EVAL.md` | reusable eval prompts or runbooks |
+| `references/` | runtime-loadable depth |
+| `references/evidence/` | persistent iteration examples |
+
+## Template
+
+```markdown
+# <Skill Name> Specification
+
+## Intent
+
+<1-2 short paragraphs>
+
+## Scope
+
+In scope:
+- ...
+
+Out of scope:
+- ...
+
+## Users And Trigger Context
+
+- Primary users:
+- Common user requests:
+- Should not trigger for:
+
+## Runtime Contract
+
+- Required first actions:
+- Required outputs:
+- Non-negotiable constraints:
+- Expected bundled files loaded at runtime:
+
+## Source And Evidence Model
+
+Authoritative sources:
+- ...
+
+Useful improvement sources:
+- positive examples:
+- negative examples:
+- commit logs/changelogs:
+- issue or PR feedback:
+- eval results:
+
+Data that must not be stored:
+- secrets
+- customer data
+- private URLs or identifiers not needed for reproduction
+
+## Reference Architecture
+
+- `SKILL.md` contains:
+- `references/` contains:
+- `references/evidence/` contains:
+- `scripts/` contains:
+- `assets/` contains:
+
+## Evaluation
+
+- Lightweight validation:
+- Deeper evaluation:
+- Holdout examples:
+- Acceptance gates:
+
+## Known Limitations
+
+- ...
+
+## Maintenance Notes
+
+- When to update `SKILL.md`:
+- When to update `SOURCES.md`:
+- When to update `EVAL.md`:
+- When to update `references/evidence/`:
+```
+
+## Design Rules
+
+1. Keep `SPEC.md` concise.
+2. Link to `SOURCES.md` or refs instead of duplicating them.
+3. Keep raw examples in `references/evidence/`.
+4. Keep sensitive data redacted.

package/src/internal-skills/skill-writer/references/structure-troubleshooting.md

@@ -0,0 +1,111 @@
+# Structure Troubleshooting
+
+Load this when the skill layout is unclear, overloaded, or drifting away from focused routing.
+
+## Over-long SKILL.md
+
+Problem: `SKILL.md` exceeds 500 lines and becomes a second encyclopedia.
+
+Fix: extract detailed material into focused `references/` files and keep `SKILL.md` as the router.
+
+## Missing Trigger Keywords
+
+Problem: the description is too vague to match user language.
+
+Fix: include the phrases users actually say.
+
+## Trigger Info In Body Instead Of Description
+
+Problem: "when to use" guidance appears only in the body, after triggering has already happened.
+
+Fix: move trigger language into `description`.
+
+## Duplicating CLAUDE.md Or AGENTS.md
+
+Problem: the skill repeats repo-wide conventions instead of adding net-new value.
+
+Fix: reference existing project docs and keep the skill focused on domain-specific behavior.
+
+## Unconditional Reference Loading
+
+Problem: the skill tells the agent to read every reference up front.
+
+Fix: add a decision table so references load only when needed.
+
+## Large References Without Navigation
+
+Problem: long reference files are hard to preview and easy to misuse.
+
+Fix: add a table of contents or split by lookup need.
+
+## Extraneous Files
+
+Problem: the skill directory accumulates user-facing docs or miscellaneous notes that do not help runtime, evaluation, or maintenance.
+
+Fix: keep only `SKILL.md`, `SPEC.md`, `SOURCES.md`, `EVAL.md`, `references/`, `scripts/`, `assets/`, and `LICENSE` when needed.
+
+## Scripts Without Documentation
+
+Problem: `SKILL.md` names a script but does not document arguments, output, or fallback behavior.
+
+Fix: document the script interface and expected output shape in `SKILL.md`.
+
+## Hardcoded Paths
+
+Problem: the skill embeds host-specific or repo-hardcoded paths.
+
+Fix: use skill-root-relative paths or established portable placeholders.
+
+## First/Second Person Descriptions
+
+Problem: the description says "I can..." or "You can use this..."
+
+Fix: write in third person so skill discovery stays consistent.
+
+## Time-Sensitive Information
+
+Problem: the skill bakes in dates or transitional logic that will quietly rot.
+
+Fix: move legacy behavior into a clearly labeled deprecated section or remove it.
+
+## Advanced Mechanics Without Justification
+
+Problem: the skill uses routing, `context: fork`, hooks, or evaluator loops because they seem sophisticated.
+
+Fix: name the shape, explain why simpler shapes were rejected, and add portability/safety notes.
+
+## Router Without Fallback
+
+Problem: the skill has multiple downstream paths but no default route or clarification step.
+
+Fix: add a fallback branch that asks one clarifying question or picks a documented safe default.
+
+## Evaluator Loop Without Stop Condition
+
+Problem: the skill says "iterate until good" with no rubric or cap.
+
+Fix: add a rubric plus a max-loop or plateau rule.
+
+## Passive Guidance In Forked Context
+
+Problem: `context: fork` is used for conventions or reference material instead of a concrete delegated task.
+
+Fix: keep passive guidance inline; use forked execution only for actionable isolated tasks.
+
+## Hook-Backed Enforcement Without Safety Notes
+
+Problem: the skill uses hooks but does not explain event scope, fallback behavior, or shell risk.
+
+Fix: add hook event scope, fallback path, and explicit security notes.
+
+## Hidden Reference File
+
+Problem: a bundled reference exists, but `SKILL.md` never tells the agent when to open it.
+
+Fix: add the file to the main router with a one-line "open when..." reason, or remove/split the file if no clear reason exists.
+
+## Generic Bucket Reference
+
+Problem: a file groups unrelated techniques under a vague name such as "patterns", "notes", or "context".
+
+Fix: split the file by lookup need and rename each leaf so the filename predicts why it should be opened.

package/src/internal-skills/skill-writer/references/synthesis-path.md

@@ -0,0 +1,119 @@
+# Synthesis Path
+
+Use this path when creating or materially changing a skill.
+
+## Output Style
+
+- Keep synthesis notes terse.
+- Prefer tables, status lists, and gap lists over narrative summaries.
+- Record decisions as `adopted`, `rejected`, or `deferred`.
+
+## Step 0: Classify
+
+Record:
+
+1. skill class
+2. primary execution shape
+3. secondary shapes, if any
+4. why simpler shapes were not enough
+
+For `integration-documentation`, cover:
+
+1. API surface and behavior contracts
+2. config/runtime options
+3. downstream use cases
+4. issues/failure modes with workarounds
+5. version or migration variance
+
+## Step 1: Collect Sources
+
+Collect from:
+
+1. Agent Skills spec and best practices
+2. similar in-repo skills
+3. upstream implementations and orchestration docs
+4. domain or library docs
+5. repo conventions and validators
+6. tests, fixtures, changelogs, and issue or PR history
+7. commit history and blame for regressions or edge cases
+8. prior `SPEC.md`, `SOURCES.md`, `EVAL.md`, and `references/evidence/`
+
+If the shape uses provider-specific mechanics, include current provider docs.
+
+## Baseline Source Pack For Skill-Authoring
+
+Require at minimum:
+
+1. local `skill-writer` runtime files
+2. Agent Skills spec and repo conventions
+3. provider docs for any provider-specific mechanic being recommended
+
+## Step 1.5: Load Example Profiles
+
+Load only the profiles you need from `references/examples/`.
+
+## Step 1.6: Expand Coverage
+
+Run targeted passes for:
+
+| Pass | Retrieve |
+|------|----------|
+| core behavior | happy path and main workflow |
+| edge behavior | failures, retries, permissions, cleanup |
+| negative behavior | false positives, reviewer concerns, bad outputs |
+| repair patterns | fixes and corrected outputs |
+| version variance | platform or release differences |
+| shape mechanics | routing, delegation, loop stops, hook constraints |
+
+Extra retrieval for advanced shapes:
+
+1. route or delegation criteria
+2. worker or handoff contracts
+3. loop stopping rules
+4. provider-specific lifecycle or security constraints
+
+## Step 2: Capture Provenance
+
+For each source, record:
+
+- source URL or path
+- trust tier
+- confidence
+- contribution
+- usage constraints
+
+Store provenance in `SOURCES.md`, not long runtime prose.
+
+## Step 3: Synthesize Decisions
+
+Map each major decision to source evidence, including:
+
+- class choice
+- shape choice
+- provider-specific mechanics
+- deferred gaps
+
+## Step 4: Enforce Depth Gates
+
+All of these must pass:
+
+1. no missing high-impact coverage dimensions
+2. partial dimensions have explicit next retrieval actions
+3. authoring or generator skills include transformed examples
+4. selected profile requirements are satisfied
+5. coverage passes are reflected in the coverage matrix
+6. stopping rationale is explicit
+7. supporting refs stay focused and directly discoverable from `SKILL.md`
+8. `SPEC.md` exists or is updated when the contract changed
+9. advanced mechanics include required contracts and justification
+10. provider-specific mechanics include portability notes
+
+## Required Output
+
+- synthesis summary
+- source inventory in `SOURCES.md`
+- decisions and rationale
+- coverage matrix
+- gaps and next retrieval actions
+- selected class and shape
+- `SPEC.md` update summary when applicable

package/src/internal-skills/skill-writer/references/workflow-mechanics/evaluator-loops.md

@@ -0,0 +1,25 @@
+# Evaluator Loops
+
+Use this workflow when a first draft benefits from critique and revision against a clear rubric.
+
+## Choose this workflow when
+
+- the skill can score output against explicit criteria
+- revision improves quality materially
+- the loop has a bounded stopping rule
+
+## Required contract
+
+1. Rubric or acceptance criteria.
+2. Maximum loop count or plateau rule.
+3. What evidence or diffs must be preserved between rounds.
+
+## Example
+
+```markdown
+1. Produce a first draft
+2. Score it against the rubric
+3. Turn failures into concrete edits
+4. Revise
+5. Stop when the rubric passes or the loop plateaus
+```

package/src/internal-skills/skill-writer/references/workflow-mechanics/orchestrator-workers.md

@@ -0,0 +1,25 @@
+# Orchestrator-Workers
+
+Use this workflow when the subtasks are not known ahead of time and must be discovered from the input.
+
+## Choose this workflow when
+
+- the orchestrator must discover work units dynamically
+- each unit can be delegated to a stable worker contract
+- a final synthesis step can merge the results
+
+## Required contract
+
+1. Worker-task schema.
+2. Worker output schema.
+3. Expansion limit or stopping rule.
+4. Final synthesis rule.
+
+## Example
+
+```markdown
+1. Inspect the request and identify work units dynamically
+2. Assign each work unit to a worker path
+3. Collect worker summaries in a fixed schema
+4. Synthesize the result
+```

package/src/internal-skills/skill-writer/references/workflow-mechanics/parallel-workflows.md

@@ -0,0 +1,21 @@
+# Parallel Workflows
+
+Use this workflow when independent subtasks can run side-by-side, or when multiple judgments improve confidence.
+
+## Choose this workflow when
+
+- work units are independent
+- latency or coverage improves with parallel execution
+- aggregation can be described clearly
+
+## Common forms
+
+1. `sectioning`: split a task into independent work units.
+2. `voting`: run multiple judgments and aggregate them.
+
+## Required contract
+
+1. Unit of parallel work.
+2. Merge or vote rule.
+3. Conflict handling.
+4. Cost or latency cap.

package/src/internal-skills/skill-writer/references/workflow-mechanics/plan-validate-execute.md

@@ -0,0 +1,26 @@
+# Plan-Validate-Execute
+
+Use this workflow when a destructive or high-stakes action should be preceded by a machine-checkable plan.
+
+## Choose this workflow when
+
+- validating the plan is safer than validating the final action
+- the task changes many files or records
+- rollback would be expensive
+
+## Required contract
+
+1. The plan artifact and schema.
+2. The validation step and source of truth.
+3. Rework rules when validation fails.
+4. Final execution and verification steps.
+
+## Example
+
+```markdown
+1. Analyze the input and generate `changes.json`
+2. Validate the plan against source of truth
+3. Revise until validation passes
+4. Execute the plan
+5. Verify the result
+```

package/src/internal-skills/skill-writer/references/workflow-mechanics/prompt-chaining.md

@@ -0,0 +1,24 @@
+# Prompt Chaining
+
+Use this workflow when the task should move through fixed ordered steps and each step makes the next one easier.
+
+## Choose this workflow when
+
+- the decomposition is known in advance
+- step order matters
+- validation between steps improves quality
+
+## Required contract
+
+1. Step order.
+2. Inputs and outputs for each step.
+3. Validation or gate points between steps.
+
+## Example
+
+```markdown
+1. Summarize the current state
+2. Propose the target state
+3. Validate the proposal against constraints
+4. Write the final document
+```

package/src/internal-skills/skill-writer/references/workflow-mechanics/routing-workflows.md

@@ -0,0 +1,28 @@
+# Routing Workflows
+
+Use this workflow when the incoming request must be classified and sent to different downstream prompts, references, scripts, or tools.
+
+## Choose this workflow when
+
+- distinct request classes benefit from specialized downstream handling
+- one broad prompt would create conflicts or false positives
+- misroutes can be detected and recovered
+
+## Required contract
+
+1. Route-selection criteria.
+2. Default route or clarification fallback.
+3. Misroute recovery.
+4. Downstream contract for each route.
+
+## Example
+
+```markdown
+1. Classify the request:
+
+   **Billing question?** -> Load `references/billing.md`
+   **Refund request?** -> Run `scripts/refund_intake.py`
+   **Technical bug?** -> Load `references/triage.md`
+
+2. If classification is uncertain, ask one clarification question before continuing.
+```

package/src/internal-skills/skill-writer/references/workflow-mechanics/validation-loops.md

@@ -0,0 +1,26 @@
+# Validation Loops
+
+Use this workflow when a validator can reliably catch mistakes before the skill should claim completion.
+
+## Choose this workflow when
+
+- a script, schema, test, or parser can catch important failures
+- the agent should fix issues immediately after each change
+- the main risk is silent drift or invalid output
+
+## Required contract
+
+1. The validator or check to run.
+2. When it runs in the workflow.
+3. What to fix before retrying.
+4. What counts as a passing state.
+
+## Example
+
+```markdown
+1. Make the change
+2. Validate immediately
+3. If validation fails, fix the issue
+4. Re-run validation
+5. Only proceed when validation passes
+```

package/src/internal-skills/skill-writer/scripts/quick_validate.py

@@ -0,0 +1,158 @@
+# /// script
+# requires-python = ">=3.12"
+# dependencies = ["pyyaml"]
+# ///
+"""
+Quick validation script for Agent Skills.
+
+Validates only mechanical SKILL.md frontmatter and identity requirements.
+Qualitative content, reference depth, routing, and prose structure belong to
+human/agent review, not this script.
+
+Usage:
+    uv run quick_validate.py <skill_directory> [--skill-class <class>] [--strict-depth]
+
+Returns exit code 0 on success, 1 on failure. Outputs JSON with validation results.
+"""
+
+import argparse
+import json
+import re
+import sys
+from pathlib import Path
+
+import yaml
+
+MAX_NAME_LENGTH = 64
+MAX_DESCRIPTION_LENGTH = 1024
+
+SKILL_CLASSES = {
+    "auto",
+    "workflow-process",
+    "integration-documentation",
+    "security-review",
+    "skill-authoring",
+    "generic",
+}
+
+
+def parse_args(argv: list[str]) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Validate agent skill frontmatter and identity fields.",
+    )
+    parser.add_argument("skill_directory")
+    parser.add_argument("--skill-class", choices=sorted(SKILL_CLASSES), default="auto")
+    parser.add_argument("--strict-depth", action="store_true")
+    return parser.parse_args(argv)
+
+
+def validate_skill(
+    skill_path: Path,
+    selected_skill_class: str = "auto",
+    strict_depth: bool = False,
+) -> tuple[bool, list[str], list[str], str]:
+    """Validate a skill directory. Returns (valid, errors, warnings, resolved_skill_class)."""
+    # Keep the flag for CLI compatibility; depth/content checks belong to review.
+    _ = strict_depth
+    errors: list[str] = []
+    warnings: list[str] = []
+
+    # Check SKILL.md exists.
+    skill_md = skill_path / "SKILL.md"
+    if not skill_md.exists():
+        return False, ["SKILL.md not found"], [], "generic"
+
+    content = skill_md.read_text()
+
+    # Check frontmatter exists and is first.
+    if not content.startswith("---"):
+        errors.append("No YAML frontmatter found (file must start with ---)")
+        return False, errors, warnings, "generic"
+
+    match = re.match(r"^---\n(.*?)\n---", content, re.DOTALL)
+    if not match:
+        errors.append("Invalid frontmatter format (missing closing ---)")
+        return False, errors, warnings, "generic"
+
+    # Parse frontmatter.
+    frontmatter_text = match.group(1)
+    try:
+        frontmatter = yaml.safe_load(frontmatter_text)
+        if not isinstance(frontmatter, dict):
+            errors.append("Frontmatter must be a YAML mapping")
+            return False, errors, warnings, "generic"
+    except yaml.YAMLError as exc:
+        errors.append(f"Invalid YAML in frontmatter: {exc}")
+        return False, errors, warnings, "generic"
+
+    # Validate frontmatter keys without hardcoding provider-specific optional fields.
+    invalid_keys = [key for key in frontmatter.keys() if not isinstance(key, str) or not key.strip()]
+    if invalid_keys:
+        errors.append("Frontmatter keys must be non-empty strings")
+
+    # Validate name.
+    if "name" not in frontmatter:
+        errors.append("Missing required field: name")
+    else:
+        name = frontmatter["name"]
+        if not isinstance(name, str):
+            errors.append(f"name must be a string, got {type(name).__name__}")
+        else:
+            name = name.strip()
+            if not name:
+                errors.append("name must not be empty")
+            elif len(name) > MAX_NAME_LENGTH:
+                errors.append(f"name is too long ({len(name)} chars, max {MAX_NAME_LENGTH})")
+            elif not re.match(r"^[a-z0-9-]+$", name):
+                errors.append(f"name '{name}' must contain only lowercase letters, digits, and hyphens")
+            elif name.startswith("-") or name.endswith("-"):
+                errors.append(f"name '{name}' must not start or end with a hyphen")
+            elif "--" in name:
+                errors.append(f"name '{name}' must not contain consecutive hyphens")
+            elif name != skill_path.name:
+                errors.append(f"name '{name}' does not match directory name '{skill_path.name}'")
+
+    # Validate description.
+    if "description" not in frontmatter:
+        errors.append("Missing required field: description")
+    else:
+        description = frontmatter["description"]
+        if not isinstance(description, str):
+            errors.append(f"description must be a string, got {type(description).__name__}")
+        else:
+            description = description.strip()
+            if not description:
+                errors.append("description must not be empty")
+            elif len(description) > MAX_DESCRIPTION_LENGTH:
+                errors.append(f"description is too long ({len(description)} chars, max {MAX_DESCRIPTION_LENGTH})")
+
+    resolved_skill_class = selected_skill_class if selected_skill_class != "auto" else "generic"
+
+    return len(errors) == 0, errors, warnings, resolved_skill_class
+
+
+def main() -> None:
+    args = parse_args(sys.argv[1:])
+    skill_path = Path(args.skill_directory).resolve()
+    if not skill_path.is_dir():
+        print(json.dumps({"valid": False, "errors": [f"Not a directory: {skill_path}"]}))
+        sys.exit(1)
+
+    valid, errors, warnings, resolved_skill_class = validate_skill(
+        skill_path,
+        selected_skill_class=args.skill_class,
+        strict_depth=args.strict_depth,
+    )
+    result = {
+        "valid": valid,
+        "skill_class": resolved_skill_class,
+        "strict_depth": args.strict_depth,
+        "errors": errors,
+        "warnings": warnings,
+    }
+    print(json.dumps(result, indent=2))
+    sys.exit(0 if valid else 1)
+
+
+if __name__ == "__main__":
+    main()