@sentry/warden 0.21.0 → 0.23.0

package/src/builtin-skills/security-review/references/github-workflows.md

@@ -0,0 +1,186 @@
+# GitHub Workflow Security Notes
+
+Use this when reviewing GitHub Actions workflows, local actions, reusable workflows, or scripts/config loaded by workflows. This reference adapts the dedicated workflow-security prior art for the broad `security-review` skill; keep findings exploit-oriented, not style-oriented.
+
+## Contents
+
+- [Review Map](#review-map)
+- [Reportable Patterns](#reportable-patterns)
+- [False-Positive Controls](#false-positive-controls)
+- [Verification Checklist](#verification-checklist)
+- [Minimal Examples](#minimal-examples)
+
+## Review Map
+
+Start with the effective execution graph:
+
+1. Identify the trigger: `pull_request`, `pull_request_target`, `workflow_run`, `workflow_dispatch`, `workflow_call`, `issue_comment`, `discussion`, `label`, `push`, `release`, or `schedule`.
+2. Mark who controls each input: fork PR code, PR title/body, branch names, changed filenames, comments, labels, discussion text, manual inputs, reusable-workflow inputs, artifacts, caches, local actions, and checked-out refs.
+3. Follow every boundary: `uses: ./.github/actions/...`, `uses: ./.github/workflows/...`, composite action steps, repo-local scripts, Makefiles, package commands, artifacts, caches, and downloaded tools.
+4. Mark privileges at the point of execution: `secrets.*`, PATs, deploy keys, registry tokens, `id-token: write`, `GITHUB_TOKEN` write scopes, release/package/deploy authority, and self-hosted runners.
+5. Report only when untrusted or caller-controlled code/data reaches privileged execution, credentials, trusted artifacts, releases/packages/deployments, or sensitive runners.
+
+In-scope files include `.github/workflows/*.yml`, `.github/workflows/*.yaml`, `.github/actions/**/action.yml`, `.github/actions/**/action.yaml`, repository-root `action.yml`/`action.yaml`, and any scripts/config files loaded by those workflows.
+
+## Reportable Patterns
+
+| Pattern | Report When | Safer Shape |
+|---------|-------------|-------------|
+| Privileged PR checkout | `pull_request_target`, privileged `workflow_run`, or similar trusted context checks out, imports, builds, tests, or executes PR-controlled refs while trusted tokens/secrets are available | Use `pull_request` for untrusted code, or keep `pull_request_target` metadata-only |
+| Expression injection | Attacker/caller-controlled `${{ }}` reaches `run:`, composite shell steps, `bash -c`, `node -e`, `python -c`, `actions/github-script`, `actions/script`, or workflow command files | Move values to `env:`, read native variables, quote/validate, and avoid interpreter strings |
+| Manual or reusable input RCE | Free-form `workflow_dispatch` or `workflow_call` input reaches release, deploy, publish, signing, PR creation, OIDC, PAT, or secret-bearing commands | Use finite input types, allowlists, argv APIs, and least privilege |
+| Comment/chatops abuse | `issue_comment`, discussion, label, or slash-command workflows execute privileged commands without a trusted actor gate, or use comment text as shell/script input | Verify owner/member/collaborator/team permission and parse arguments as data |
+| Approval TOCTOU | Maintainer approval occurs, then the workflow re-resolves `pull_request.head.sha`, `head_ref`, or PR refs at run time before privileged checkout/execution | Pin the exact SHA approved by the maintainer, or require re-approval after every push |
+| Reusable/local action trust crossing | Caller grants secrets/write scopes while a callee or local composite action executes caller-controlled inputs or PR-controlled files | Pass narrow secrets, define callee permissions, validate inputs, keep local actions on trusted code |
+| Artifact/cache trust crossing | Privileged `workflow_run`, release, or deploy job executes or trusts artifacts/caches produced by untrusted PR jobs | Treat artifacts/caches as untrusted data; validate, sign, or partition trust scopes |
+| Artifact credential leak | `upload-artifact` uploads the workspace/root, `.git/`, home credential files, Docker/npm config, or similar after credentials were written | Upload only build outputs; set `persist-credentials: false`; exclude credential paths |
+| Self-hosted runner exposure | PR-reachable or comment-triggered jobs run untrusted code on persistent, internal, signing, deploy, or otherwise sensitive self-hosted runners | Keep untrusted code on GitHub-hosted runners or require a strong approval gate |
+| Mutable action supply chain | Third-party `uses: owner/action@tag`, branch, partial SHA, or mutable reusable workflow runs in a job with secrets, OIDC, write token, release, deploy, package, or signing power | Pin third-party actions and reusable workflows to a 40-character commit SHA |
+| AI agent config poisoning | Privileged workflows run coding/review agents on PR-controlled checkouts or instruction files such as `AGENTS.md`, `CLAUDE.md`, or Copilot instructions | Run agents in unprivileged PR context, protect instruction files, and avoid write/secrets in poisoned contexts |
+
+### Expression Injection Sources
+
+Treat these as untrusted when the trigger is externally reachable, manually triggerable, or callable:
+
+- PR title/body, issue title/body, comment body, review body, discussion title/body, label names, branch names, commit messages, changed filenames, and changed-file lists.
+- `inputs.*` and `github.event.inputs.*` from `workflow_dispatch`.
+- `inputs.*` from `workflow_call`, including values passed through visible caller workflows.
+- Action outputs or env vars derived from the values above.
+
+Usually not injectable by themselves: PR numbers, numeric IDs, full commit SHAs, booleans, base-repository constants, and hardcoded shell-safe `choice` inputs. Re-check them if later code reinterprets the value as shell, JavaScript, Python, package-manager flags, or another code-like language.
+
+Dangerous sinks include:
+
+- `run: echo "${{ github.event.pull_request.title }}"`
+- `actions/github-script` or `actions/script` `script:` bodies containing `${{ github.event.* }}` or `${{ inputs.* }}`
+- `echo "key=${{ github.event.comment.body }}" >> $GITHUB_OUTPUT`, `$GITHUB_ENV`, `$GITHUB_STEP_SUMMARY`, or `$GITHUB_PATH`
+- `npx semver -i ${{ inputs.bump }} "$CURRENT"`, `gh pr create --fill ${{ inputs.pr_options }}`, `docker build -t ${{ inputs.tag }}`, `git checkout ${{ inputs.ref }}`
+- Composite action shell steps that interpolate `${{ inputs.* }}` from an externally reachable caller
+
+### Privileged PR Context
+
+High-signal indicators:
+
+- `on: pull_request_target` plus `actions/checkout` with `ref: ${{ github.event.pull_request.head.sha }}`, `github.head_ref`, `github.event.pull_request.head.ref`, `repository: ${{ github.event.pull_request.head.repo.full_name }}`, `refs/pull/...`, or custom `git fetch` of PR refs.
+- Build/test/package commands after PR checkout: `npm install`, `pnpm install`, `npm test`, `pip install`, `tox`, `pytest`, `make`, `cargo`, `go test`, `bundle`, `gradle`, `mvn`.
+- Local actions, scripts, Makefiles, package lifecycle hooks, or config files loaded from the PR checkout.
+- `persist-credentials` omitted or true before untrusted code runs.
+- Write scopes, secrets, OIDC, package publishing, release creation, deployments, or agent write access available in the same job.
+
+Do not report `pull_request_target` that only labels, comments, or reads metadata without checking out or loading PR-controlled files.
+
+### Reusable Workflows, Artifacts, Caches, And Credentials
+
+Trace cross-file flows before reporting:
+
+- `workflow_call` callers that pass secrets/write permissions into a callee that executes caller-controlled inputs.
+- Reusable workflows that reference `secrets.X` without declaring `X` under `on.workflow_call.secrets`, other than `GITHUB_TOKEN`; this hides the secret surface and pressures callers into `secrets: inherit`.
+- Reusable workflows without top-level or job-level `permissions:` when the callee needs a narrower scope than callers commonly grant.
+- `workflow_run` jobs that download artifacts from untrusted PR workflows and execute scripts, import code, publish packages, or make trusted comments without validation.
+- Caches shared from untrusted PR jobs into privileged jobs, including eviction-and-replace poisoning of expected cache keys.
+- `actions/upload-artifact` whose `path:` includes `.`, `./`, `${{ github.workspace }}`, `.git/`, `~/.docker/config.json`, `~/.npmrc`, `~/.gitconfig`, `~/.aws/credentials`, or other credential-bearing paths after checkout/login/setup steps.
+- `id-token: write` where untrusted refs can satisfy visible cloud OIDC trust policies.
+
+Permissions and secrets are amplifiers. Tie them to the untrusted execution or leak path.
+
+### Mutable Action References
+
+Report mutable third-party actions only when job privilege makes compromise security-relevant. CVE-2025-30066 (`tj-actions/changed-files`) and related 2025 supply-chain incidents showed that tag rewrites can leak secrets at scale.
+
+Severity guide:
+
+| Shape | Severity |
+|-------|----------|
+| Mutable third-party ref in package publishing, release signing, protected-branch push, production deploy, or token-minting job | high |
+| Mutable third-party ref with secrets, OIDC, or non-trivial write-scoped `GITHUB_TOKEN` | medium |
+| Pinned action that downloads and executes mutable remote scripts in a privileged job | medium, or high when the downloaded payload runs inside the privileged step |
+| Mutable third-party ref in public read-only CI with no secrets and no write scopes | no finding unless adjacent to another traced workflow risk |
+
+First-party `actions/*` and `github/*` actions on version tags are not findings by themselves. Same-repo or vendored actions are not third-party supply-chain findings, but can still be unsafe if they are loaded from PR-controlled checkouts.
+
+## False-Positive Controls
+
+- Broad `permissions:` alone is not a vulnerability. Report it only as part of untrusted execution, credential exposure, artifact trust, or privileged side effect.
+- Plain `pull_request` normally has restricted token and no base secrets for forks. Trace downstream artifacts/caches before escalating.
+- `${{ }}` in `if:`, ordinary `with:`, or `env:` is not a sink unless the receiving action or a later shell/script interprets it as code. `actions/github-script` `with: script:` is a code sink.
+- `env:` is only safe when the later shell/script uses native variables with quoting or validation. `echo '${{ env.BODY }}'` is still expression expansion.
+- Hardcoded `choice`, `boolean`, `number`, and `environment` workflow inputs are usually safe when used only in `if:`, ordinary `with:`, or safely quoted `env:` contexts.
+- Comment/body parsing is not a bug unless it triggers meaningful execution or privileged state change.
+- `CONTRIBUTOR` is not equivalent to `MEMBER`, `OWNER`, or `COLLABORATOR` for chatops authorization.
+- `persist-credentials: false` reduces `.git/config` token theft, but does not protect unrelated secrets, PATs, OIDC, or registry credentials.
+- Do not invent external action internals. If source is unavailable, report the unresolved trust assumption as medium confidence at most.
+
+## Verification Checklist
+
+Before reporting:
+
+1. Confirm the trigger can be reached by the attacker or lower-privileged caller you name.
+2. Identify the exact attacker-controlled or caller-controlled value.
+3. Identify the sink: shell/script execution, local action, package lifecycle hook, artifact/cache trust, credential-bearing upload, mutable action, or sensitive runner.
+4. Follow local actions, reusable workflows, scripts, package commands, and workflow-produced artifacts/caches.
+5. Confirm secrets, token scopes, OIDC, deploy/release/package authority, or runner sensitivity at the sink.
+6. Check actor gates, branch/fork guards, finite input types, SHA pinning, `persist-credentials: false`, artifact path narrowing, and cloud trust-policy constraints.
+7. Anchor the finding to the changed workflow line, and name the crossed boundary plus concrete impact.
+
+## Minimal Examples
+
+**Report: privileged PR checkout**
+
+```yaml
+on: pull_request_target
+permissions: write-all
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+      - run: pnpm install
+      - run: pnpm test
+```
+
+Risk: fork code controls package scripts while the job has trusted-repository permissions.
+
+**Report: github-script injection**
+
+```yaml
+on: issues
+jobs:
+  comment:
+    steps:
+      - uses: actions/github-script@v7
+        with:
+          script: |
+            github.rest.issues.createComment({
+              body: `hello ${{ github.event.issue.title }}`
+            })
+```
+
+Risk: the issue title is expanded into JavaScript before `actions/github-script` runs.
+
+**Report: workspace artifact leak**
+
+```yaml
+steps:
+  - uses: actions/checkout@v4
+  - uses: actions/upload-artifact@v4
+    with:
+      path: .
+```
+
+Risk: `.git/config` can contain persisted checkout credentials, and public-repo artifacts can expose them.
+
+**Do not report: metadata-only target workflow**
+
+```yaml
+on: pull_request_target
+permissions:
+  pull-requests: write
+jobs:
+  label:
+    steps:
+      - run: gh pr edit "$PR" --add-label needs-review
+```
+
+No PR-controlled code or text reaches execution. Broad token scope alone is not enough.

package/src/builtin-skills/security-review/references/javascript-typescript.md

@@ -0,0 +1,69 @@
+# JavaScript And TypeScript Security Notes
+
+Use this when reviewing JavaScript, TypeScript, Node, React, Next.js, or browser code. These examples refine the core skill; they do not add new reporting scope.
+
+## Server-Side Entry Points
+
+- Next.js route handlers, Server Actions, API routes, tRPC/RPC handlers, Express/Fastify/Koa routes, webhook handlers, queue consumers, and CLI/service functions can be security boundaries.
+- Treat Server Actions as callable server entry points. UI visibility, hidden form fields, and client components are not authorization.
+- Next.js `middleware.ts` is not enough proof of authorization for sensitive mutations. Verify handler-level auth and resource-level authorization.
+- For service handlers, do not trust caller-controlled headers such as `x-user-id`, `x-org-id`, `x-forwarded-*`, or internal-only flags unless a trusted gateway verifies them.
+
+## High-Signal Patterns
+
+| Pattern | Vulnerable | Safer |
+|---------|------------|-------|
+| Server Action authz | `"use server"` mutation trusts `userId`, `teamId`, `role`, hidden fields, or form args | Load session in the action, scope by server-side tenant, enforce permission |
+| SQL injection | `prisma.$queryRawUnsafe(\`...${id}\`)`, string-built SQL, concatenated `where` clauses | Tagged templates, query parameters, ORM filters scoped to the authenticated tenant |
+| Command injection | `exec("git " + branch)`, `spawn(cmd, args, {shell: true})` with user input | `execFile`/`spawn` with fixed binary, fixed argument positions, and strict allowlists |
+| XSS | `innerHTML`, `dangerouslySetInnerHTML`, `unsafeHTML`, unsafe Markdown HTML, inline script JSON with user data | Text rendering, vetted sanitizer, escaping `<` in inline JSON, no dangerous URL schemes |
+| SSRF | `fetch(req.query.url)`, image/preview/proxy fetchers using user URLs, redirect-following after first-hop validation | Exact host allowlist, private-IP blocking, DNS rebinding defenses, redirect revalidation or manual redirects |
+| Open redirect | `redirect(searchParams.get("next"))`, prefix/substring URL checks in login or OAuth flows | Relative-path allowlist or exact origin/path allowlist after URL normalization |
+| Path traversal | `path.join(root, userPath)` without realpath containment, archive extraction by entry name | Normalize and resolve real paths, verify containment, generate server-side filenames |
+| Webhook forgery | State-changing webhook parses JSON before verifying signature or skips timestamp/replay checks | Verify raw body signature, timestamp freshness, replay/idempotency, and provider secret before side effects |
+| Secrets exposure | Secrets in client components, `NEXT_PUBLIC_*`, serialized props, logs, or error responses | Server-only reads, redacted logs, safe error messages, no hardcoded production fallback |
+
+## False-Positive Controls
+
+- React text interpolation escapes by default. Report only escape hatches or dangerous URL/script contexts.
+- Prisma/Drizzle/Knex query builders can parameterize values. Verify the specific API before reporting SQL injection.
+- `crypto.randomUUID()` and `crypto.getRandomValues()` are suitable for security randomness; `Math.random()` is not.
+- `jsonwebtoken.verify` can be safe when algorithms, issuer/audience, expiry, and key selection are pinned appropriately.
+- DOMPurify or equivalent sanitizers can mitigate HTML injection when configured for the target context.
+
+## Minimal Examples
+
+**Report: cross-tenant lookup**
+
+```ts
+const invoice = await db.invoice.findUnique({ where: { id: params.invoiceId } });
+```
+
+Require: server-derived tenant scope such as `accountId: session.accountId`, plus permission checks.
+
+**Report: Server Action trusts caller fields**
+
+```ts
+"use server";
+export async function setRole(userId: string, role: string) {
+  await db.user.update({ where: { id: userId }, data: { role } });
+}
+```
+
+Require: load session in the action and prove caller can mutate that tenant user.
+
+**Report: inline JSON script breakout**
+
+```tsx
+<script dangerouslySetInnerHTML={{ __html: `window.__DATA__=${JSON.stringify(data)}` }} />
+```
+
+Risk: `</script>` breakout. Require: escape `<` or `</script` before embedding.
+
+**Do not report: parameterized query**
+
+```ts
+await db.$queryRaw`SELECT * FROM invoices WHERE id = ${invoiceId} AND account_id = ${accountId}`;
+```
+
+This is not SQL injection if the tagged template parameterizes values and `accountId` is trusted from the authenticated session.

package/src/builtin-skills/security-review/references/python.md

@@ -0,0 +1,70 @@
+# Python Security Notes
+
+Use this when reviewing Python, Django, Flask, FastAPI, Celery, or Python service code. These examples refine the core skill; they do not add new reporting scope.
+
+## Server-Side Entry Points
+
+- Django views, DRF viewsets, Flask/FastAPI routes, GraphQL resolvers, webhook handlers, Celery tasks, management commands, and service-layer functions can cross trust boundaries.
+- For background jobs and tasks, verify the caller, queue, payload signing, tenant context, and idempotency before assuming input is trusted.
+- Decorators can prove authentication only if they wrap the effective handler. Still verify object-level authorization.
+- DRF/FastAPI auth dependencies prove identity, not ownership of route params or body IDs.
+
+## High-Signal Patterns
+
+| Pattern | Vulnerable | Safer |
+|---------|------------|-------|
+| SQL injection | `cursor.execute(f"...{request.GET['q']}...")`, string-built raw SQL | Parameterized queries, ORM filters, strict enum allowlists for identifiers |
+| Command injection | `os.system`, `subprocess.run(..., shell=True)`, shell strings with request data | `subprocess.run([fixed_binary, fixed_arg])`, strict allowlists, no shell |
+| Path traversal | `open(base / request.args["name"])`, `send_file(user_path)`, unsafe archive extraction | `Path.resolve()` containment checks, generated filenames, safe storage APIs |
+| SSRF | `requests.get(request.GET["url"])`, preview/proxy fetchers, redirect-following after first-hop validation | Exact host allowlist, block private/link-local IPs, disable or revalidate redirects |
+| Open redirect | `redirect(request.GET["next"])`, login/callback redirects with weak host checks | Relative-path allowlist or framework helper with exact allowed hosts |
+| Unsafe deserialization | `pickle.loads`, `yaml.load` without `SafeLoader`, model/job loaders on uploaded data | JSON or typed schemas, `yaml.safe_load`, signed trusted artifacts only |
+| XSS | Jinja/Django `|safe`, `Markup`, disabled autoescape, raw HTML from user content | Autoescaping, vetted sanitizer, context-correct escaping |
+| Authz bypass | `Model.objects.get(id=request.GET["id"])` on tenant data | Scope by authenticated user/org/account and enforce permissions before returning or mutating |
+| Task trust confusion | Celery task mutates `invoice_id`, `user_id`, or `account_id` queued from a request without rechecking scope | Pass server-derived actor/tenant context and re-check before mutation |
+| Secrets exposure | Logging tokens, cookies, auth headers, signed URLs, stack traces, or env secrets | Redacted logging, generic errors, server-only secret access |
+
+## False-Positive Controls
+
+- Django and SQLAlchemy ORM filters usually parameterize values. Raw SQL and string-built identifiers need closer review.
+- Django and Jinja autoescape ordinary template interpolation by default. Report only unsafe filters, raw HTML, or disabled autoescape.
+- `secrets` and `os.urandom` are suitable for security randomness; `random` is not.
+- `yaml.safe_load` is the safe default for untrusted YAML; `yaml.load` may still be safe only when an explicit safe loader is used.
+- `Path.resolve()` containment checks can mitigate traversal when they compare the resolved child against the resolved allowed root.
+
+## Minimal Examples
+
+**Report: object-level authorization bypass**
+
+```python
+invoice = Invoice.objects.get(id=request.GET["invoice_id"])
+return JsonResponse({"total": invoice.total, "email": invoice.customer.email})
+```
+
+Require: authenticated account/org scope and permission before returning sensitive data.
+
+**Report: task loses tenant context**
+
+```python
+@shared_task
+def approve_invoice(invoice_id: str):
+    Invoice.objects.filter(id=invoice_id).update(status="approved")
+```
+
+Require: reload trusted actor/tenant context and enforce permission.
+
+**Report: unsafe uploaded state**
+
+```python
+state = pickle.loads(base64.b64decode(request.POST["state"]))
+```
+
+Risk: unsafe deserialization. Require: trusted, signature-verified artifact before deserialization.
+
+**Do not report: scoped ORM query**
+
+```python
+invoice = Invoice.objects.get(id=invoice_id, account_id=request.user.account_id)
+```
+
+This is not an authorization bypass if `request.user.account_id` is trusted and the caller is authenticated for that account.

package/src/internal-skills/skill-writer/EVAL.md

@@ -0,0 +1,124 @@
+# Skill Writer Eval Prompts
+
+Use these prompts when deeper evaluation matters (high-risk, regression tracking, or explicit request).
+These are optional guidance artifacts, not required outputs for every skill.
+
+## Advanced Shape Selection Eval
+
+```text
+Update or create the requested skill and select the simplest adequate execution shape.
+
+For every scenario:
+- name the skill class
+- name the primary execution shape
+- explain why simpler shapes were rejected only if you choose an advanced shape
+- add portability notes if you use provider-specific mechanics
+
+Scenarios:
+
+1. "Create a skill that triages billing, refund, and technical-support requests into distinct downstream guidance."
+Expected shape signal: `router`
+
+2. "Create a skill that gives SQL naming conventions and query-style preferences for this repo."
+Expected shape signal: `inline-guidance` or `reference-backed-expert`, not router or subagent
+
+3. "Create a skill that runs a self-contained investigation in isolated context and returns a concise findings summary."
+Expected shape signal: `subagent-fork`
+
+4. "Create a skill that must block risky Bash commands with deterministic validation before they run."
+Expected shape signal: `hook-backed`
+
+5. "Create a skill that drafts a proposal, critiques it against a rubric, and revises until it passes."
+Expected shape signal: `evaluator-optimizer`
+
+Hard fail if:
+- no explicit execution shape is selected
+- an advanced shape is chosen without justification
+- a provider-specific mechanic is used without portability notes
+- a passive guidance skill is incorrectly put into `context: fork`
+- a bundled reference file is added without a direct routing entry in `SKILL.md`
+```
+
+## Integration/Documentation Depth Eval
+
+```text
+Synthesize a new skill named `pi-agent-integration-eval` for working with `@mariozechner/pi-agent-core` as a consumer in downstream libraries.
+
+Primary objective: produce a non-surface-level integration skill that covers API surface, known issues/workarounds, and common real-world use cases.
+
+Scope:
+- Source root: `<pi-mono-root>/packages/agent`
+- This is for USING Pi in another library, not editing Pi internals.
+
+Mandatory source retrieval:
+- README, CHANGELOG
+- `src/index.ts`, `src/agent.ts`, `src/agent-loop.ts`, `src/types.ts`, `src/proxy.ts`
+- `test/agent.test.ts`, `test/agent-loop.test.ts`
+- In-repo usage scan for key APIs (for example Agent, agentLoop, streamProxy, convertToLlm, transformContext, steer, followUp, continue)
+
+Required depth artifacts:
+- focused references covering API surface, at least 6 concrete downstream use cases, and at least 8 failure modes with fixes/workarounds
+- a focused integration-patterns or examples reference with happy path, robust variant, and anti-pattern + correction
+
+Depth gates (hard fail if missing):
+- Coverage matrix includes: API surface, options/config, runtime lifecycle, event semantics, queue semantics, failure modes, version variance, downstream usage patterns.
+- Any partial coverage includes explicit next retrieval actions.
+- Qualitative depth rubric includes pass/fail for API/workaround/use-case/gap handling.
+- Run validator and report output.
+
+Output sections:
+1) Summary
+2) Changes Made
+3) Validation Results
+4) Open Gaps
+```
+
+## Pass/Fail Rubric
+
+Pass only if all required artifacts exist and have the requested depth.
+Fail if API mapping is partial, workaround guidance is shallow, or use cases are generic and not actionable.
+Fail if completion is claimed with unresolved high-impact gaps and no next retrieval actions.
+Fail if the selected execution shape is more complex than necessary and the writeup does not justify it.
+Fail if bundled reference files are hidden behind vague bucket docs instead of clear routed leaves.
+
+## Optional Deep-Eval Pattern
+
+When you need stronger confidence, run this sequence:
+
+1. Use a fixed prompt set (positives + negatives).
+2. Capture deterministic traces (`codex exec --json`).
+3. Apply rubric/schema checks where practical (`--output-schema`).
+4. Compare baseline vs candidate and report deltas.
+
+## Isolated Eval Runbook
+
+Run the eval in a temporary isolated workspace (copy of repo in `/tmp`):
+
+```bash
+EVAL_DIR=/tmp/sentry-skills-eval-run
+rm -rf "$EVAL_DIR"
+mkdir -p "$EVAL_DIR"
+rsync -a "<repo-root>/"/ "$EVAL_DIR"/
+
+codex exec \
+  --ephemeral \
+  --full-auto \
+  --sandbox workspace-write \
+  --skip-git-repo-check \
+  --add-dir "<pi-mono-root>" \
+  -C "$EVAL_DIR" \
+  "$(cat <eval-prompt-file>)"
+```
+
+Where `<eval-prompt-file>` contains the exact eval prompt from this file.
+
+Validate the generated skill output:
+
+**Requires**: The `uv` CLI for python package management, install guide at https://docs.astral.sh/uv/getting-started/installation/
+
+```bash
+uv run "<skill-writer-root>/scripts/quick_validate.py" \
+  /tmp/sentry-skills-eval-run/.agents/skills/pi-agent-integration-eval \
+  --skill-class integration-documentation \
+  --strict-depth
+```

package/src/internal-skills/skill-writer/SKILL.md

@@ -0,0 +1,159 @@
+---
+name: skill-writer
+description: Create, synthesize, and iteratively improve agent skills following the Agent Skills specification. Use when asked to "create a skill", "write a skill", "synthesize sources into a skill", "improve a skill from positive/negative examples", "update a skill", or "maintain skill docs and registration". Handles source capture, depth gates, authoring, registration, and validation.
+---
+
+# Skill Writer
+
+Use this as the single canonical workflow for skill creation and improvement.
+Primary success condition: maximize high-value input coverage before authoring while minimizing wasted runtime tokens.
+
+Load only the path(s) required for the task. `SKILL.md` is the primary router: every bundled reference file should have a direct "open when..." reason here.
+
+## Core Workflow References
+
+| Open when you need to... | Read |
+|--------------------------|------|
+| choose the minimum workflow path for create, update, iterate, or research-first work | `references/mode-selection.md` |
+| choose the simplest adequate execution shape before deciding files | `references/execution-shapes.md` |
+| apply writing constraints for depth, concision, and portability | `references/design-principles.md` |
+| decide what belongs in `SKILL.md`, `references/`, `SPEC.md`, or subfolders | `references/reference-architecture.md` |
+| create or update the maintenance contract for a skill | `references/spec-template.md` |
+| find missing high-signal sources, including history and regressions | `references/source-discovery.md` |
+| run the full synthesis pass with depth gates and source capture | `references/synthesis-path.md` |
+| author or update `SKILL.md`, `SPEC.md`, and supporting files | `references/authoring-path.md` |
+| improve trigger language and false-positive/false-negative behavior | `references/description-optimization.md` |
+| iterate from positive, negative, or fix examples | `references/iteration-path.md` |
+| store persistent working and holdout examples for future revisions | `references/iteration-evidence.md` |
+| choose a response template, schema, or output contract | `references/output-contracts.md` |
+| troubleshoot overloaded layouts, hidden refs, or other structure failures | `references/structure-troubleshooting.md` |
+| verify a risky, disputed, or explicitly requested change | `references/evaluation-path.md` |
+| register the skill and run final validation checks | `references/registration-validation.md` |
+
+## Artifact Layout References
+
+| Open when you need to... | Read |
+|--------------------------|------|
+| keep the whole skill inline in one coherent `SKILL.md` | `references/artifact-layouts/inline-skill-layout.md` |
+| split optional deep knowledge into focused routed references | `references/artifact-layouts/reference-backed-skill-layout.md` |
+| add scripts for deterministic automation or validation | `references/artifact-layouts/script-backed-skill-layout.md` |
+| define a skill that is usually invoked with explicit arguments | `references/artifact-layouts/argument-driven-skill-layout.md` |
+| ship reusable templates, schemas, or other static assets | `references/artifact-layouts/asset-template-skill-layout.md` |
+
+## Workflow Mechanic References
+
+| Open when you need to... | Read |
+|--------------------------|------|
+| break a task into fixed ordered steps | `references/workflow-mechanics/prompt-chaining.md` |
+| classify requests and route them to different downstream paths | `references/workflow-mechanics/routing-workflows.md` |
+| split independent work into parallel units or votes | `references/workflow-mechanics/parallel-workflows.md` |
+| discover work units dynamically and coordinate worker outputs | `references/workflow-mechanics/orchestrator-workers.md` |
+| critique and revise output against a rubric | `references/workflow-mechanics/evaluator-loops.md` |
+| run validate-fix-repeat checks during authoring or execution | `references/workflow-mechanics/validation-loops.md` |
+| validate a plan before executing a risky action | `references/workflow-mechanics/plan-validate-execute.md` |
+
+## Claude Code References
+
+| Open when you need to... | Read |
+|--------------------------|------|
+| use Claude-specific frontmatter or invocation controls | `references/claude-code/frontmatter-and-invocation.md` |
+| use Claude argument fields or substitution variables | `references/claude-code/argument-substitutions.md` |
+| build a skill that runs in isolated `context: fork` | `references/claude-code/subagent-fork-skills.md` |
+| build a skill that uses Claude hooks for deterministic enforcement | `references/claude-code/hook-backed-skills.md` |
+| use Claude shell preprocessing for dynamic context injection | `references/claude-code/dynamic-context.md` |
+
+## Example Profiles
+
+| Open when you need to... | Read |
+|--------------------------|------|
+| see the expected depth for a documentation-heavy skill | `references/examples/documentation-skill.md` |
+| see the expected depth for a security-review skill | `references/examples/security-review-skill.md` |
+| see the expected depth for a workflow-process skill | `references/examples/workflow-process-skill.md` |
+| see what a good routed skill looks like | `references/examples/router-skill.md` |
+| see what a good evaluator-loop skill looks like | `references/examples/evaluator-loop-skill.md` |
+| see what a good subagent-fork skill looks like | `references/examples/subagent-fork-skill.md` |
+| see what a good hook-backed skill looks like | `references/examples/hook-backed-skill.md` |
+
+## Step 1: Resolve target, path, and shape
+
+1. Resolve the intended operation (`create`, `update`, `synthesize`, `iterate`) and inspect workspace prior art before choosing where files belong.
+2. Choose the target skill root from observed conventions. If the canonical location is still unclear after inspection, ask one direct question before editing files.
+3. Read `references/mode-selection.md` to choose the minimum required workflow paths.
+4. Read `references/execution-shapes.md` to choose the primary execution shape.
+5. Default to the simplest adequate shape. If selecting a more complex shape, record why simpler shapes were rejected.
+6. Load only the exact artifact-layout, workflow-mechanic, and provider-specific leaf files required by that shape.
+7. Record portability implications before using provider-specific mechanics.
+
+## Step 2: Run synthesis when needed
+
+Read `references/synthesis-path.md`.
+
+1. Use this path for new skills, material changes, and research-first planning.
+2. Collect and score relevant sources with provenance.
+3. Read `references/source-discovery.md` when source material is thin, stale, or ambiguous.
+4. Produce source-backed decisions and coverage/gap status, including the class and execution-shape choice.
+5. Load example profiles only when they add concrete depth for the selected class or shape.
+6. If the skill uses provider-specific mechanics, include current official provider docs and capture usage constraints.
+7. Do not move to authoring until depth gates pass.
+
+## Step 3: Run iteration first when improving from outcomes/examples
+
+Read `references/iteration-path.md` first when selected path includes `iteration` (for example operation `iterate`).
+
+1. Capture and anonymize examples with provenance.
+2. Read `references/iteration-evidence.md` when examples should persist beyond the current turn.
+3. Re-evaluate skill behavior against working and holdout slices.
+4. Propose improvements from positive/negative/fix evidence.
+5. Carry concrete behavior deltas into authoring.
+
+Skip this step when selected path does not include `iteration`.
+
+## Step 4: Author or update skill artifacts
+
+Read `references/authoring-path.md`.
+
+1. Write or update `SKILL.md` in imperative voice with trigger-rich description.
+2. Keep `SKILL.md` as the runtime router, not an encyclopedia.
+3. Read `references/reference-architecture.md` before adding bulk instructions or new reference files.
+4. Create or update `SPEC.md` using `references/spec-template.md` when creating a new skill or materially changing its contract.
+5. Create focused reference files, subfolders, scripts, and assets only when each one has a clear "open when..." reason.
+6. If you add a bundled reference file, add a direct routing entry for it in this `SKILL.md`.
+7. Prefer checklists, tables, templates, and input/output examples over explanatory prose.
+8. Follow only the specific artifact-layout, workflow-mechanic, Claude-specific, and output-contract references selected for this skill.
+9. For advanced execution shapes, add the required routing, delegation, or safety contracts before considering the skill complete.
+10. For authoring/generator skills, include transformed examples in references:
+   - happy-path
+   - secure/robust variant
+   - anti-pattern + corrected version
+
+## Step 5: Optimize description quality
+
+Read `references/description-optimization.md`.
+
+1. Validate should-trigger and should-not-trigger query sets.
+2. Reduce false positives and false negatives with targeted description edits.
+3. Keep trigger language generic across providers unless the skill is intentionally provider-specific.
+
+## Step 6: Evaluate only when needed
+
+1. Read `references/evaluation-path.md` only when the user asks for evaluation, the change is high-risk, or the architecture choice is non-obvious.
+2. If you run evaluation, start with the lightweight qualitative check.
+3. Run deeper evals only when requested or risk warrants it.
+4. Record outcomes and unresolved risks when evaluation is run.
+
+## Step 7: Register and validate
+
+Read `references/registration-validation.md`.
+
+1. Apply repository registration steps for the active layout you verified in the workspace.
+2. Run quick validation with strict depth gates.
+3. Reject shallow outputs that fail depth gates or required artifact checks.
+
+## Output format
+
+Return:
+
+1. `Summary`
+2. `Changes Made`
+3. `Validation Results`
+4. `Open Gaps`