@evalops/maestro 0.10.49 → 0.10.51

package/skills/incident-triage/SKILL.md

@@ -25,11 +25,29 @@ Use this skill to turn scattered incident evidence into a concise operator plan.
 ## Workflow
 
 1. Confirm the symptom, start time, affected surface, and urgency.
-2. Load `reference/triage.md` for the detailed timeline and mitigation checklist.
+2. Load learned guidelines (see below) and `reference/triage.md` for the detailed timeline and mitigation checklist.
 3. Gather only scoped evidence from granted tools and local files. Keep customer data and internal handles out of the normal answer.
 4. Build a timeline with known, inferred, and unknown entries separated.
 5. Identify the likely owner, immediate mitigation, verification signal, and follow-up issue.
 6. End with current state, blast radius, next action, and what evidence was withheld or unavailable.
+7. Record what you learned (see below) so the next incident starts from it.
+
+## Learned Guidelines
+
+This skill accumulates alert-type knowledge across runs in
+`~/.maestro/skills/incident-triage/guidelines.md`.
+
+- **At the start of a run**, read that file if it exists and treat its entries as
+  priors to verify — which tools, dashboards, owners, and mitigations a given
+  alert type needed last time. Confirm they still hold; do not trust them
+  blindly.
+- **At the end of a run**, append one concise entry mapping the alert type or
+  symptom you handled to the tools/interfaces, likely owner, and mitigation that
+  actually worked. Keep entries short and free of secrets and customer data.
+
+Programmatic callers can use `loadLearnedGuidelines` /
+`appendLearnedGuideline` / `formatLearnedGuidelinesForPrompt` from
+the main Maestro package (`src/skills/learned-guidelines.ts`).
 
 ## Toolbox
 

package/skills/install-code-review/SKILL.md

@@ -0,0 +1,69 @@
+---
+name: install-code-review
+description: Install automated Maestro code review in a GitHub repository's CI — add a pull_request workflow that runs `maestro exec` and posts a merge-readiness review as a PR comment. Use when the user wants to set up automated PR review, wire Maestro into CI/CD, or "install code review".
+license: Complete terms in LICENSE.txt
+compatibility: "Maestro skill packages with write/bash access to a checked-out GitHub repository."
+allowed-tools:
+  - read
+  - write
+  - bash
+builtin-tools:
+  - read
+  - write
+mode: build
+metadata:
+  version: "0.1.0"
+  category: evalops-operations
+  artifactSchema: evalops.maestro.skill.install_code_review.v1
+---
+
+# Install Code Review
+
+Wire Maestro into a repository's CI so every pull request gets an automated
+merge-readiness review. Use the published generator exports
+(`buildMaestroReviewWorkflow` / `writeMaestroReviewWorkflow`) so installs work
+in arbitrary repositories, not just in the Maestro monorepo. In this monorepo,
+the same implementation also lives at
+`packages/github-agent/src/workflows/maestro-review-workflow.ts` and is
+re-exported by `@evalops/github-agent`.
+
+## Workflow
+
+1. **Confirm the target.** Verify you are in a Git repository with a GitHub
+   remote, and that `.github/workflows/` is the right place (some repos vendor
+   workflows elsewhere). Confirm the default branch.
+2. **Choose options.** Ask only what you cannot infer: the model provider
+   (default inferred from the provider secret/env name, otherwise `anthropic`),
+   provider API key secret name (default inferred from the provider runtime env
+   var), and, if the user wants a pinned model, the model id. Pass
+   `maestroPackage` from `MAESTRO_PACKAGE_NAME` when set, otherwise use the
+   default `maestro` package, and Node to 20. For custom providers, set the runtime
+   API-key environment variable explicitly if it is not the provider id
+   uppercased with `_API_KEY` appended.
+3. **Write the workflow.** Generate `.github/workflows/maestro-review.yml` from
+   the published generator (do not hand-roll the YAML — use the generator so the
+   file is deterministic and reviews cleanly). The workflow triggers on
+   `pull_request` (opened, synchronize, reopened), runs `maestro exec` with the
+   `pr-review` skill, and posts the result with `gh pr comment`.
+4. **Document the secret.** Tell the user exactly which repository secret to add
+   (the API key), which runtime env var the workflow maps it to, and that the
+   workflow uses the built-in `GITHUB_TOKEN` for the PR comment. Do not write any
+   secret value into the repo.
+5. **Land it.** Create a branch, commit the workflow, and open a PR titled
+   "Install Maestro code review" — never commit the workflow directly to the
+   default branch. Summarize what was added and the one secret the user must
+   set for it to run.
+
+## Output
+
+Report the workflow path written, the required secret name, the trigger events,
+and the PR you opened. If the repo already has a `maestro-review.yml`, show the
+diff and ask before overwriting.
+
+## Guardrails
+
+- Request only the permissions the job needs: `contents: read` and
+  `pull-requests: write`.
+- Never embed an API key in the workflow or commit a token.
+- Keep the review step non-interactive (`maestro exec`); never assume a TTY in
+  CI.

package/skills/security-review/SKILL.md

@@ -0,0 +1,68 @@
+---
+name: security-review
+description: Correctness-first, depth-first security audit of a single repository using STRIDE, OWASP Top 10, the OWASP LLM Top 10, and supply-chain analysis. Use when the user asks for a security review, a vulnerability audit, a threat model, or a pre-release security pass.
+license: Complete terms in LICENSE.txt
+compatibility: "Maestro skill packages with read/search/bash access to a single checked-out repository."
+allowed-tools:
+  - read
+  - search
+  - bash
+builtin-tools:
+  - read
+  - search
+mode: review
+isolatedContext: true
+metadata:
+  version: "0.1.0"
+  category: evalops-operations
+  artifactSchema: evalops.maestro.skill.security_review.v1
+---
+
+# Security Review
+
+Produce a depth-first security audit of one repository. Depth beats breadth: a
+small number of substantiated, exploitable findings is worth more than a long
+list of pattern matches. Every finding must trace a concrete path from an
+untrusted input to a dangerous sink, with `file:line` evidence. Do not report a
+finding you cannot substantiate by reading the code.
+
+## Workflow
+
+1. **Scope.** Identify the languages, frameworks, entry points (HTTP routes,
+   CLI, queue consumers, webhooks), trust boundaries, and where untrusted input
+   enters. Note authentication and authorization surfaces.
+2. **Threat model (STRIDE).** Enumerate threats per trust boundary using the
+   STRIDE categories in `reference/rubric.md`. This drives where you look; it is
+   not itself a finding list.
+3. **Trace data flow.** For each candidate issue, follow the data from source
+   (untrusted input) to sink (query, exec, deserialization, file path, template,
+   model prompt, response). Confirm there is no effective sanitization on the
+   path before recording a finding.
+4. **Apply the catalogs.** Check against OWASP Top 10, the OWASP LLM Top 10 (for
+   any LLM/agent surface), and supply-chain risks — all detailed in
+   `reference/rubric.md`. Load that file when you need the full checklists. If a
+   catalog check surfaces a new candidate, send it back through step 3 and report
+   it only after the source→sink trace is substantiated.
+5. **Rate and write up.** Assign severity with the rubric. Each finding gets:
+   title, severity, location (`file:line`), the source→sink path, impact, and a
+   concrete, minimal fix.
+6. **Report.** Lead with findings ordered by severity. If you find nothing
+   exploitable, say so plainly and name the residual risk you could not rule out
+   (e.g. dependencies you did not audit, code paths you could not reach).
+
+## Findings Artifact
+
+Emit findings as `evalops.maestro.skill.security_review.v1`: a list of findings,
+each with `id`, `title`, `severity` (critical | high | medium | low | info),
+`category` (stride/owasp/owasp-llm/supply-chain), `location`, `path`, `impact`,
+and `fix`. Include a short `summary` with counts by severity and the scope you
+actually covered.
+
+## Discipline
+
+- No speculative findings. If you cannot show the vulnerable path, it is not a
+  finding — at most a note in residual risk.
+- Prefer reading the real code over running scanners; if you run a scanner (e.g.
+  `semgrep`, `npm audit`), treat its output as leads to verify, not findings.
+- Never exfiltrate secrets you discover. Report the location and the exposure,
+  redact the value.

package/skills/security-review/reference/rubric.md

@@ -0,0 +1,85 @@
+# Security Review Rubric
+
+Load this file when you need the full checklists and the severity rubric. It is
+intentionally kept out of the main skill body so the agent only pays for it when
+a review is actually underway.
+
+## Severity
+
+| Severity | Meaning |
+| --- | --- |
+| critical | Remotely exploitable with no auth, or trivial privilege escalation / RCE / full data exfiltration. Fix before release. |
+| high | Exploitable with some precondition (auth, specific input, race). Real impact on confidentiality, integrity, or availability. |
+| medium | Exploitable only with significant preconditions, or limited blast radius. Defense-in-depth gap. |
+| low | Hardening issue, weak default, or risk that requires an unlikely chain. |
+| info | Observation worth recording; not exploitable on its own. |
+
+Rate by realistic impact and exploitability of the **traced path**, not by the
+category. A SQL string concatenation behind three layers of validation is not a
+critical.
+
+## STRIDE (threat model)
+
+Walk each trust boundary and ask what each category enables:
+
+- **Spoofing** — can an actor claim another identity? (auth, session, tokens,
+  signature verification, TLS validation)
+- **Tampering** — can data or code be modified in transit or at rest? (integrity
+  checks, signed artifacts, mass-assignment, mutable shared state)
+- **Repudiation** — can an action be denied? (audit logging, tamper-evident
+  logs)
+- **Information disclosure** — can secrets or data leak? (error messages, debug
+  endpoints, directory listing, verbose logs, IDOR)
+- **Denial of service** — can an actor exhaust resources? (unbounded input,
+  regex catastrophic backtracking, zip bombs, missing rate limits)
+- **Elevation of privilege** — can an actor gain capabilities? (authorization
+  gaps, path traversal, deserialization, command injection)
+
+## OWASP Top 10 (2021)
+
+1. Broken access control (IDOR, missing authz on object/function, path
+   traversal, forced browsing).
+2. Cryptographic failures (plaintext secrets, weak/legacy algorithms, missing
+   TLS verification, predictable randomness).
+3. Injection (SQL/NoSQL, OS command, LDAP, XPath, template injection, XSS).
+4. Insecure design (missing threat model, unsafe-by-default flows).
+5. Security misconfiguration (debug on, default creds, permissive CORS, open S3,
+   verbose errors).
+6. Vulnerable and outdated components (see Supply Chain below).
+7. Identification and authentication failures (weak session handling, credential
+   stuffing exposure, missing MFA on sensitive flows).
+8. Software and data integrity failures (unsigned updates, insecure
+   deserialization, CI/CD trust).
+9. Security logging and monitoring failures (no audit trail on sensitive
+   actions).
+10. Server-side request forgery (unvalidated outbound URLs / fetchers).
+
+## OWASP LLM Top 10
+
+Apply to any prompt, agent, tool-calling, or RAG surface:
+
+1. Prompt injection (direct and indirect, including via retrieved/tool content).
+2. Sensitive information disclosure (secrets or PII leaking into prompts,
+   completions, or logs).
+3. Supply chain (untrusted models, plugins, datasets).
+4. Data and model poisoning.
+5. Improper output handling (model output flowing unsanitized into a sink — SQL,
+   shell, HTML, downstream tool calls).
+6. Excessive agency (tools with more authority than the task needs; missing
+   human approval on destructive actions).
+7. System prompt leakage (secrets or trust-bearing instructions in the system
+   prompt).
+8. Vector and embedding weaknesses (injection or poisoning via the retrieval
+   store).
+9. Misinformation / overreliance (acting on unverified model claims).
+10. Unbounded consumption (token/cost exhaustion, denial of wallet).
+
+## Supply Chain
+
+- Pinned, integrity-checked dependencies (lockfiles, hashes) vs. floating
+  ranges.
+- Known-vulnerable versions (`npm audit`, `pip-audit`, `osv-scanner` as leads).
+- Install/build scripts that execute on `install` (postinstall hooks).
+- Typosquat / dependency-confusion risk on internal package names.
+- CI/CD: secrets in workflow logs, unpinned actions, write tokens on PRs from
+  forks.